Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Constructivism and instructional design : an exploration using an asynchronous online nominal group technique Janes, Diane P. 2005

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2005-105119.pdf [ 16.08MB ]
Metadata
JSON: 831-1.0055625.json
JSON-LD: 831-1.0055625-ld.json
RDF/XML (Pretty): 831-1.0055625-rdf.xml
RDF/JSON: 831-1.0055625-rdf.json
Turtle: 831-1.0055625-turtle.txt
N-Triples: 831-1.0055625-rdf-ntriples.txt
Original Record: 831-1.0055625-source.json
Full Text
831-1.0055625-fulltext.txt
Citation
831-1.0055625.ris

Full Text

CONSTRUCTIVISM A N D INSTRUCTIONAL DESIGN: A N EXPLORATION USING A N ASYNCHRONOUS ONLINE N O M I N A L GROUP TECHNIQUE by DIANE P. JANES M.Ed., Memorial University of Newfoundland, 1993 B.A. , Memorial University of Newfoundland, 1981 A THESIS SUBMITTED IN PARTIAL F U L F I L L M E N T OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE F A C U L T Y OF G R A D U A T E STUDIES (Educational Studies) THE UNIVERSITY OF BRITISH C O L U M B I A September, 2005 © Diane P. Janes, 2005 A B S T R A C T The purpose of this study was to identify the principal tenets/presuppositions of constructivism as a theory of learning, and to suggest processes or instructional approaches that flow from this theory for the development of online learning. Using a consensus building methodology, the Nominal Group Technique (NGT) wholly online, the intention was to seek the consensus of experts/practitioners in the field in order to identify an agreed-upon theory of constructivism. Possible approaches to instruction and delineation of the role of the instructor that flow from the theory were explored. In all, two idea statements generated by the participant/experts garnered a 100% consensus within the study. An additional 33 idea statements were deemed to have reached a consensus of between 80-99%. The following description, emerging from the study, uses the language and concepts employed by the participant-experts in the study: Constructivism encourages and values the personal understanding/knowledge construction by the learner and the design of learning is student-centred. Understanding that there are multiple ways that knowledge is absorbed and constructed, there are a number of conditions that should be encouraged that were felt to be essential to constructivism. Learning is iterative and lifelong; it fosters active, higher-level thinking and reflection; it is social in nature. It builds upon shared negotiated meaning and upon the learner's interests. It is situated within realistic authentic tasks and complex problem solving, based on and found in the world outside of the 'classroom'. Instructors share the authority and assist i i students in adopting intellectual responsibility for their own learning. Instructors are flexible in their support of student expressions of varying levels of knowledge while critically merging the learner's knowledge with disciplinary insights and personal experiences. Assessment should include learner reflection on their goals and accomplishments and avoid overly narrow measurement of skills or knowledge without a complete context of interpretation. In addition, recommendations are included for further research and suggestions are made for the use of the NGT as a research methodology. i i i \ T A B L E OF CONTENTS A B S T R A C T i i LIST OF T A B L E S ix LIST OF FIGURES ., xi A C K N O W L E D G E M E N T S xii CHAPTER 1 INTRODUCTION 1 Background and Statement of Problem 1 Overall Research Focus and Questions 3 Significance of the Study 4 Limitations of the Study 5 Outline of the Study 6 CHAPTER 2 LITERATURE REVIEW 8 The Evolution of Learning Theory 8 Behavioural Learning Theory 8 Cognitive Learning Theory 9 Constructivist Learning Theory 10 Constructivism 10 Constructivism as a Continuum • 20 Cognitive Constructivism , 22 Social Constructivism 23 Radical Constructivism 24 From Learning Theory to Instructional Theory 25 Constructivism and the Instructional Design Framework 27 Developments in Online Learning 37 Integrating Constructivism, Instructional Design, and Online Learning 40 iv Summary 42 CHAPTER 3 DESIGN A N D METHODOLOGY 44 Conducting Research Online 44 Structured Group Processes - Consensus Building 53 The Decision-Making Process 56 The Delphi Procedure 57 S earch Conference Methodology 57 Systems Thinking Using Group Model Building (GMB) 58 Nominal Group Technique (NGT) 59 The Opportunity 69 Conclusion of the Methodology Decisions 71 Plan of Action for the Online NGT 72 Site and Setting 72 Group Size 73 The Participant-Experts 74 The Facilitator 75 Schedule of Events 75 NGT Questions for the Participant-Experts 77 Definition and Measurement of Agreement 78 Data Analysis 79 Ethical and Safety Issues 80 CHAPTER 4 THE NGT PROCESS A N D ITS EFFICACY ONLINE 82 Platform Choices 82 Setting up the Online NGT 86 Data Forum 88 v Thesis Site Backup 88 Participant-Expert Selection 88 Participant Demographics 90 Implementation of Round 1 91 Analysis Activity Between Rounds 1 and 2 92 Implementation of Round 2 94 Analysis Activity Between Rounds 2 and 3 96 Implementation of Round 3 97 Analysis Activity Between Rounds 3 and 4 98 Implementation of Round 4 98 End of Process Survey 99 CHAPTER 5 D A T A A N A L Y S I S 101 Round 1 Data 101 Round 2 Data 115 Round 3 Data 118 Data Analysis Layout 119 Consensus 100% 120 Consensus 90-99% 123 Consensus 80-89% 128 Consensus 70-79% 134 Consensus 60-69%. 139 Consensus Less Than 60% 142 Consensus: Constructivist Learning Activities 144 Summary 148 v i CHAPTER 6 DISCUSSION 152 Round 4 Data 152 Comments on Round 3 Vote 153 Consensus on Consensus Level 157 Affirmation of the Consensus Vote 158 Summary 159 Online N G T 160 Participant Commitment 162 Group Size of Online NGT ; 162 Round 1 Brainstorming 163 Time Line of the Online NGT 164 The Technology - Platform/Tool Choices 166 Summary of Online NGT Process 169 CHAPTER 7 S U M M A R Y , CONCLUSIONS A N D RECOMMENDATIONS 170 Summary of the Study 170 Conclusions 171 Conclusion One 171 Conclusion Two 172 Conclusion Three 172 Conclusion Four 173 Conclusion Five 173 Recommendations from NGT 173 Preliminary Preparation 174 Participant Commitment. 174 Group Size of Online NGT 174 v i i Round 1 Brainstorming 175 Time Line of the Online NGT 175 The Technology - Platform/Tool Choices 176 Further Study of the Online NGT . 176 Further Study of Constructivism and Online Learning 176 Summary 177 Rationale for NGT 177 Toward a Theory of Constructivism 178 Philosophy or Theory? 180 REFERENCES 182 APPENDIX A DRAFT LETTER OF CONSENT 199 APPENDIX B DELPHI A N D NGT DIAGRAMS 202 APPENDIX C TYPES OF ASYNCHONOUS GROUP C O M M U N I C A T I O N 204 APPENDIX D E M A I L CORRESPONDENCE WITH DR. K . H O F F M A N N 207 APPENDIX E E M A I L CONTACT WITH STUDY PARTICIPANTS 210 APPENDIX F E M A I L WITH SUPERVISOR RE: ROUND 2 REVISION 225 APPENDIX G S U R V E Y INSTRUMENT 228 APPENDIX H IDEA STATEMENTS GENERATED IN ROUND 1 257 APPENDIX I ROUND 3 F U L L D A T A SET 266 APPENDIX J SCREEN CAPTURES OF STUDY WEBSITE 272 v i i i LIST OF T A B L E S Table 1: Three Categories of Learning Theory 11 Table 2: Jonassen's Suggested "Model" for Designing C L E 30 Table 3: Wilson's Postmodern Instructional Design (ID) in Traditional Elements 35 Table 4: C A A Stages in the NGT Study 66 Table 5: Comparison of Platform/Tool Features Considered for the NGT ..85 Table 6: Participant Selection Process 90 Table 7: Demographics of Participant-Experts 91 Table 8: Participant-expert Groups and Category/Idea Statement Breakdown 95 Table 9: Round 2A Idea Statements 103 Table 10: Round 2B Idea Statements 110 Table 11: Breakdown of Participation in Round 2.2 116 Table 12: Idea Statements Generated in Round 2.2 117 Table 13.1: Results from Round 3 online vote - Question 1 (100%) 121 Table 13.2: Results from Round 3 online vote - Question 1 (90-99%) 124 Table 13.3: Results from Round 3 online vote - Question 1 (80-89%) 129 Table 13.4: Results from Round 3 online vote - Question 1 (70-79%) 135 Table 13.5: Results from Round 3 online vote - Question 1 (60-69%) 139 Table 13.6: Results from Round 3 online vote - Question 1 (Less than 60%) 142 Table 13.7: Results from Round 3 online vote - Question 1 (Learning Activities) 145 Table 14.1: Results from Round 3 online vote - Question 2 (100%) 122 Table 14.2: Results from Round 3 online vote - Question 2 (90-99%) 126 ix Table 14.3: Results from Round 3 online vote - Question 2 (80-89%) 131 Table 14.4: Results from Round 3 online vote - Question 2 (70-79%) 137 Table 14.5: Results from Round 3 online vote - Question 2 (60-69%) 140 Table 14.6: Results from Round 3 online vote - Question 2 (Less than 60%) 143 Table 14.7: Results from Round 3 online vote - Question 2 (Learning Activities) 146 Table 15.1: Results from Round 3 online vote - Question 3 (100%) 123 Table 15.2: Results from Round 3 online vote - Question 3 (90-99%) 127 Table 15.3: Results from Round 3 online vote - Question 3 (80-89%) 133 Table 15.4: Results from Round 3 online vote - Question 3 (70-79%) T38 Table 15.5: Results from Round 3 online vote - Question 3 (60-69%) 141 Table 15.6: Results from Round 3 online vote - Question 3 (Less than 60%) 144 Table 15.7: Results from Round 3 online vote - Question 3 (Learning Activities) 148 Table 16: Participant Opinion of Online NGT as a Discussion Vehicle 161 Table 17: Participant Engagement over the Four-Month NGT Process 165 Table 18: Participant Opinions Regarding Access to WebCT Platform 166 Table 19: Participant Opinions of WebCT as an Online Research Platform 167 Table 20: Participant Experience in Using Hosted Survey ....168 X LIST OF FIGURES Figure 1: Homepage to Online Discussion Website for Study xi A C K N O W L E D G E M E N T S This journey is never undertaken alone. I have been lucky in that I have been surrounded by so many who have contributed to my trek. I want to acknowledge the support and advice of the following: • My committee, Dr. Tom Sork, Dr. Jean Barman and Dr. A.W. (Tony) Bates. I could not have chosen a better team to work with. • Dr. Mary F. Kennedy, who got me started down this road and who is always there when I was not sure if I needed to go left or right! • My parents, June Janes and Willis Janes, my sister Sharon Janes Tobin and her family and my brother Keith Janes and his family. • The 38 participants in the study, who stayed with me no matter what was happening in their professional and personal lives. It was appreciated. • Friends and colleagues, especially at the University of Saskatchewan, Extension Division; the University of British Columbia, Distance Education & Technology; and Memorial University of Newfoundland. Thank you all. This thesis is dedicated to my two grandmothers: Mildred Coish Steele (1906 - 1997) and Maude Whalen Janes (1913 - 2002) Two of the most intelligent, creative and strong Newfoundland women I have had the privilege to know and love. I miss you both. xii CHAPTER 1 INTRODUCTION Background and Statement of Problem Of all of the "learning theories" that I have explored in my work as an instructional developer, I am drawn to those that ".. .can serve as a guide in the design of learning environments" (Murphy, 1997b). Learning environments that bring the individual beyond the individual to the social and cultural issues and that have an impact on our learning and being. I am also drawn to the concept of multiplicity, which for me has significant implications for learning and teaching. Multiplicity is an overriding concept in constructivism, as it characterizes not only the epistemological and theoretical underpinnings but in many ways how the theory is expressed. Yet while there are many variants of constructivism and theories have advanced into so many fantastic routes and avenues, it remains for me a series of common themes that permit the creation of principles, instructional models and general characteristics. In following the writings of Fosnot (1992), I suggest that, while constructivism is a thriving and debated theory of knowing and coming to know, "it is not as yet, a well-documented theory of teaching" (p. 169). This question of a theory of constructivism and approaches to creating online instruction that flow from that theory, remains, as does the role of the online instructor in those processes (Hsiao, 1996). Boudourides (2003) observes that According to von Glasersfeld (1990, p. 37), all good teachers know that guidance which they give to students "necessarily remains tentative and cannot ever 1 approach absolute determination," because in constructivism there is always more than one solution to a problem and different solutions might be approached from different perspectives. A l l these imply that knowledge cannot simply be transferred by means of words. Von Glasersfeld holds that Verbally explaining a problem does not lead to understanding, unless the concepts the listener has associated with the linguistic components of the explanation are compatible with those the explainer has in mind. Hence it is essential that the teacher have an adequate model of the conceptual network within which the student assimilates what he or she is being told. Without such a model as [a] basis, teaching is likely to remain a hit-or-miss affair. From the constructivist perspective, 'learning' is the product of self-organization (Boudourides, 2003, p. 11) An online Delphi by Gabriel, Ostridge, and Doiron (2003) identified four elements in their study of best practice in online teaching and learning. The first three were ". . . [1] Structuring learning activities, [2] forming learning groups and [3] facilitating interactions" (p. 6). Their fourth element was the role of the instructor. In their conclusion they noted that .. .the role of the instructor is still emerging, as increasing numbers of university and college faculty offer courses online...this role becomes more clearly defined as teachers develop an understanding about instructional design in online environments... (p. 7) 2 And that is the issue. Educators are being asked to jump into this way of designing online instruction and teaching which is unfamiliar and is, without a doubt, a difficult one. The use of constructivism in the online classroom is also a call for teachers, researchers, academics and future scholars to move toward and begin to advocate, mold and design what might become online constructivist theories of instruction. This probing of possible consensus about/concerning online constructivist instruction is my thesis research area. Does constructivism offer approaches to online instruction that can guide teachers, instructional designers and students? If it does, what do these approaches look like? Can they be reproduced to ensure that learners have the greatest opportunity to 'create and build' on their own online learning? Is constructivism a theory of learning that cannot be 'taught' in the traditional sense? The methodology that I used, a consensus building technique, has not to date been often applied electronically. Yet, as Anderson and Kanuka (2003) point out, "the Net, with its capacity to support a variety of synchronous and asynchronous as well as group and individual communication modes, is an ideal environment to support existing and experiment with new varieties of consensus data collection" (p. 121). Overall Research Focus and Questions The purpose of this study was to explore the principle tenets/presuppositions of constructivism as a theory of learning, and to suggest processes or instructional approaches that flow from this theory, in the development of online learning. To explore this issue, I proposed the following research questions: 3 • What are regarded as the primary features, characteristics, elements of constructivism as a theory of learning? • What approaches or procedures to instruction in relation to online learning seem to flow from constructivism as it is currently conceived? • Given these characteristics and procedures, what is the role of the instructor? Significance of the Study In undertaking this study I hoped it would make three different kinds of contributions: First, a contribution to theory - No one that I am aware of has made an effort to arrive at a consensus on what constitutes constructivism. That is, what are the key features of constructivism as identified and agreed upon by experts and practitioners, and as it relates to our understanding of constructivism as a theory in relation to online learning? Second, a contribution to practice - Constructivism has become a key concept used in literature, conference presentations and other professional arenas with respect to online learning. Yet I have seen no evidence that there is any consensus on what the implications are for online learning, and for the development of courses, and for engaging students in online courses. My study contributes to changes in practice. If people believe that constructivism is an important theoretical base for online learning then my research will contribute to a better, deeper understanding of how to apply the theory. Fundamentally, what I am trying to do is to integrate instructional design approaches with 4 the actual tenets of constructivist theory and hence diminish the current dichotomy between the theory and how people develop online instruction. Third, a contribution to online research methods and techniques - 1 wanted to further the exploration of online versions of various research methods that have not had much previous application. With this in mind I used an online Nominal Group Technique (NGT), and examined its utility as an online research process. Limitations of the Study There are a number of limitations, which I must clarify before proceeding. As this was my first foray into the NGT area I felt I had to prepare myself for this experience. To do this, I pilot tested, where possible each phase of the research as I proceeded. This took some additional time, but I felt it was important to the success of the final process. NGT has not often been attempted in a wholly online environment and there was little to guide my work in this research. Given this, the pilot testing of the 'rounds' or cycles traditionally found in an NGT, was all the more important. I attempted to do this during the preliminary stages of the work. This research was limited to a one-time NGT. While the literature indicates that such a scenario has the potential to garner strong results, it is a small study, with limited potential to be generalized. However it does offer opportunities for transferability. The ideas that I generate have the potential to be transferable to other settings. Other researchers may discover that the implications or the extensions of constructivism into 5 instruction found here are consistent with the kind of work that they do and the beliefs that they have about quality of instruction. Finally, participation in this research was limited to English speaking participant-experts who were able to navigate a visually-oriented website. Outline of the Study Chapter Two contains a review of the literature on the evolution of constructivism, constructivism and instructional design, instructional theory, and developments in online learning. Chapter Three explores the advantages and disadvantages of conducting online research, and structured group processes including consensus-building methods. In addition, it reviews the choices I made in arriving at the use of the nominal group technique and the exemplars that underpinned my decisions. It outlines the process I followed for the N G T and specific choices regarding group size, the participant-expert criteria, the use and role of the facilitator and the schedule of events for the online NGT. Chapter Four documents my work in setting up the NGT. It reviews my choices of platform, my selection of the participant-experts, my contact with them, their consent to participate, and the activities associated with getting the NGT underway. It is also the chapter that describes the NGT process as it progressed from the first day interaction with the participant-experts, to the synthesis of their brainstorming, the voting processes and the final discussions and consensus. It documents the online use of NGT as a research methodology. 6 Chapter Five analyzes the results of the brainstorming and the voting process, documents the qualitative discussions and charts the group's arrival to consensus. Chapter Six ties the results of the NGT process and the final consensus of the participant-experts to the research questions and the literature in the field. It relates what I found in my research that either challenges or builds on the current literature. Chapter Seven reaches conclusions on the value of the consensus and the utility of the NGT as a research approach. It ends with recommendations for future research in this area from both the perspective of constructivism and the use of NGT as an online research methodology. 7 CHAPTER 2 LITERATURE REVIEW This chapter reviews the literature in the area of learning theory and specifically the development of constructivism, instructional theory, and instructional design frameworks. It also explores current developments in online learning. The Evolution of Learning Theory Constructivism has its educational roots in learning theory. There are three acknowledged branches of learning theory: behaviourism, cognitivism, and constructivism (Doolittle, 1999a; Bransford, Brown, & Cocking, 1999; Crocco, 2001; Steffe & Kieren, 1994; Young, 2003; Gulati, 2004). Behaviourism, as epitomized in North America by B. F. Skinner in the 1940s, was supplanted in the 1960s by cognitive learning theory, which devolved into and was to some extent supplanted in the late 1980s by constructivist learning theory. Brief sketches of these bodies of theory follow. Behavioural Learning Theory Behavioural learning theory is concerned with learner performance - performance is accepted as the only valid evidence that learning has occurred. The basis of behaviourism is B. F. Skinner's premise of operant conditioning (Skinner, 1938). Learning is believed to be a conditioned response to a stimulus. The stimulus is provided, a response is elicited, and the response is reinforced, either positively or negatively. The behavioural approach is often informally referred to as reinforcement 8 theory, because reinforcement plays such a key role in the process. In behavioural learning theory, learning is reduced to conditioned behaviours. What actually occurs within the brain of the learner is beyond the interest of behaviourists. Despite its simplistic view of learning, in certain circumstances it works, but it leaves many questions unanswered about how learners actually learn. Cognitive Learning Theory Cognitivism is an internal orientation to learning. Cognitive learning theorists are more interested in changes in what the learners know, and the structure of their knowledge as stored within the brain. They emphasize how learners process new information, incorporate it into existing information/knowledge, and how and where it is stored, remembered and recalled. Cognitive learning theories are frequently referred to as information-processing theories. They emphasize complex intellectual processes such as thinking, language development or acquisition, and problem-solving. Essential to cognitivism is the concept of meaningful learning. Learners make knowledge their own in relation to their understanding and in relation to its meaning. According to Smith and Ragan (1999) for them "recent cognitive learning theorists have concentrated primarily upon the later stages of information processing. Specifically they have conjectured upon the structures and processes surrounding encoding information into long-term memory from working memory, and retrieval of information from long-term memory into working memory" (p. 22). 9 Constructivist Learning Theory Constructivism is an internal orientation to learning. It is deemed to be a new theory of learning, but in actuality it has been around in an older guise - that of discovery learning - for decades. Constructivism assumes that knowledge is individually constructed and/or socially constructed by learners, based on their interpretations of learning and world experiences. Therefore, knowledge cannot be transmitted: rather, learners become engaged in meaning-making. As Seels (1989) notes on constructivism: "...learning occurs because personal knowledge is constructed by an active and self-regulated learner ... who reflects on theoretical explanations" (p. 11). Knowledge, and in fact the learning process itself, is rooted in the learner's unique interpretation of the world. Seels (1989) provides us with a visual reference (see Table 1) that briefly summarizes these three categories of learning theory in relation to learning and instruction (p. 14). Constructivism So what is constructivism and how did it evolve? To some extent, this question has not resulted in one discrete answer. Definitions concerning constructivism are often over simplistic or inconsistent. I will attempt to bring you to a definition by way of the theories and philosophies that have been suggested by the literature. Several theorists, namely Jean Piaget, Lev S. Vygotsky, Jerome Bruner, and Ernst von Glasersfeld, share similar visions and influences in terms of constructivism (See 10 Fosnot, 1996, and others). But constructivism has not easily come to an agreed upon place in the literature. Table 1 Three Categories of Learning Theory Learning Theory Learning & Instruction Behavioural Cognitive Constructivist Learning Is... Change in behaviour due to conditioning Programming of new rule for information processing Personal discovery based on insight Types of Learning Discrimination, Generalization, Chaining, Association Short-term sensory storage, Short-term memory, Long-term memory Problem-solving Key Concept Reinforcement Elaboration Autotelic principle (intrinsic motivation) Source: Seels, B. (1989). The instructional design movement in educational technology. Educational Technology, 29(5), 11-15. ERIC Document Reproduction Service No. EJ 395 529. This is most likely because it is viewed from two perspectives: that of a philosophy of learning, and that of a learning theory. (For the purposes of this study, I am using the distinctions of philosopher John Passmore (1967) who separated philosophy into three conceptions - wisdom, ideology and critical inquiry. Theories on the other hand have a shared set of characteristics, according to Suppe (1977). They are meant to explain something or help us understand the way things are within a certain sphere of influence. They are often a cluster of concepts organized together to form a whole, with 11 not all concepts precisely defined. They usually come with some way of connecting evidence to observation). It would be wise to caution here that this tension between theory and philosophy is not one that I have invented but is one that is found in the current literature. It is not always easy to distinguish between the two. To do so depends a great deal on what you think a theory is - is it a demanding definition or is it one that is more of an abstract notion? According to Doolittle (1999a), "...Constructivism is a theory of learning that has roots in both philosophy and psychology" fl|15). Walker and Lambert (1995) go on to note that for them "Constructivism is a theory of learning, and it is also a theory of knowing. It is an epistemological concept that draws from a variety of fields, including philosophy, psychology, and science" (p. 1). Constructivism "has become de rigueur in educational circles and ... stems from a long and respected tradition in cognitive psychology, especially the writings of Dewey, Vygotsky, and Piaget" (Danielson, 1996, p. 23). Essentially, the center of a definition of constructivism has learners actively constructing their own knowledge and meaning from their experiences. (Fosnot, 1996; Steffe & Gale, 1995). This core has roots that extend back through many years and many philosophers, including Dewey (1938), Hegel (1807/1949), Kant (1781/1946), and Vico (1725/1968). Jonassen (1991b) adds Descartes to the early roots of constructivism, postulating that he ".. . believed that the mind stands apart and operates independently of the body, which is a different sort of entity. He posited great powers to the mind, but was unable to say what the mind really does" (p. 7). 12 The following is a precis of the work of several of the philosophers and theorists who have been purported in the literature to have contributed to modern constructivism. I have placed the list in order of their birth and life span, in order to show how the ideas progressed and were often built on each previous thinker's work. Vico (1668- 1744) • Concluded that humans cannot rationally know the real world; but they do know the world of their own constructions i.e. their experimental world • First teach a child to be creative, thereby nurturing his or her ingenium, and let critical judgment evolve later. • Students should be taught synthetically instead of analytically. In other words, they should be taught to make connections instead of divisions. The categorical method so central to western logic dulls the ingenium that is central to practical wisdom. [Ingenium is defined as ingenuity, inventiveness, mother wit] (Vico, 1988, p. 96) • Whoever postulates that in reality postulates this physical thing will be true only for whoever has made them just as geometrical [proofs] are true for men just because men make them. (Vico, 1988, p. 96). This final comment reflects Vico's verum-factum principal: you can best know a thing if you make the thing. We can know geometry because we have created geometry. We cannot completely know the physical world because we did not make the physical world (Kaminski, 2003, If 19). Rousseau (1712- 1778) • Tried to grasp the emotional and passionate side of man which he felt was left out of most previous philosophical thinking • His ideas about education included minimizing the importance of book learning. • Recommended a child's emotions should be educated before reason • Placed a special emphasis on learning by experience • Learning is done through trial and error, experimentation through concrete medium (Encyclopedia of Philosophy 1967, p. 221) Kant (1724- 1804) • Time and space are imposed on experience by the human mind and that the 'reality' beyond our experience is not conceivable to us • Influenced by Hume; ideas found in the work of James and Dewey • Between theory and practice, no matter how complete the theory may be, a middle term that provides a connection and transition is necessary. (Kant, 1983, p. 61) 13 • Knowledge is no more than a means to an end [...] unless knowledge can be put in the service of appropriate ends, it cannot truly benefit individuals or society (Kant, 1983, p. 2) Hegel (1770- 1831) • Belongs to period of 'German idealism' that followed Kant • As a teacher disliked traditional didactic forms of instruction (Butler & Seiler, 1984, p. 199) • "Education to independence demands that young people should be accustomed early to consult their own sense of propriety and their own reason" (Mackenzie, 1909, p. 175) • "To regard study as mere receptivity and memory work is to have a most incomplete view of what instruction means." (Mackenzie, 1909, p. 167) • Fundamentally Hegel viewed education and learning as 'experiential' (Hegel, 1977, p. 55) Froebel (1782- 1852) • Sought to encourage the creation of educational environments that involved practical work and the direct use of materials. Through engaging with the world, understanding unfolds • Play is significant - both through and with creative activity, learners become aware of their place in the world • Originated the 'kindergarten system' • The teacher/student relationship should be one of equality, not authority. (Lilley, 1967, p. 56) • Self-expression is through life experiences. (Lilley, 1967, p. 49) • Teachers should lead students through discovery, not dictate what should be learned. (Lilley, 1967, p. 51) James (1842- 1910) • Contains seeds of pragmatism and phenomenology • Talks to Teachers (1899) - It was here that he put forward his now famous theory on learning by doing. This was to heavily influence John Dewey, and the future of educational theory through to Kolb and others • A teacher "must start with the native tendencies, and enlarge the pupil's entire passive and active experience. He must ply him with new objects and stimuli, and make him taste the fruits of his behavior, so that now that whole context of remembered experience is what shall determine his conduct when he gets the stimulus, and not the bare immediate impression" (James, 1899, p. 45) • "Begin with the line of his [the student's] native interests, and offer him objects that have some immediate connection with these" (James, 1899, p. 63) • Next, step by step, connect with these first objects and experiences the later objects and ideas which you wish to instill. Associate the new with the old in some natural and telling way, so that the interest, being shed along from point to point, finally suffuses the entire system of objects of thought" (James, 1889, p. 63) 14 • "If the topic be highly abstract, show its nature by concrete examples" (James, 1889, p. 72) • "The art of remembering is the art of thinking" . . . and so "when we wish to fix a new thing in either our own mind or a pupil's, our conscious effort should not be so much to impress and retain it as to connect it with something else already there. The connecting is the thinking; and if we attend clearly to the connection, the connected thing will certainly be likely to remain within recall" (James, 1899, p. 87) • Children should not be expected to learn by rote. Their experiences must be turned into useful and habitual behaviour through action. The learner must listen, but then take notes, experiment, write essays, measure, consult and apply. He recommends learning through work and the creation of real things or dealings with real people in a shop, to give you educational experiences beyond mere theory. He was a firm advocate of vocationally oriented schools and work-based learning. Dewey (1859- 1952) • More inclined to draw inferences about ontology than Piaget • Regarding the development of cognitive structures Piaget and Dewey use different terms but are in fundamental agreement as they both stress the fact that all knowledge is based on sensorimotor experience and they have a similar conception of'abstraction' (von Glasersfeld, 1999, Tf2) • Only true education comes through the stimulation of the child's powers by the demands of the social situations in which he finds himself (Dewey, 1897, p. 78) • Education being a social process, the school is simply that form of community; education, therefore, is a process of living and not a preparation for future living; the school must represent present life (Dewey, 1897, p. 78) • The active side precedes the passive in the development of the child nature • Like Vygotsky, viewed the mind and its formation as a communal process • Important that education not be the teaching of mere fact, but that the skills and knowledge learned be integrated fully into their lives • Founder of the philosophical school of Pragmatism (along with Charles Sanders Peirce and William James) - learning by doing sprang from this school of thought • Emphasis on problem solving and critical thinking rather than memorization • Dewey worked strongly from Hegel Montessori (1870- 1951) • Argued for the development of teacher training, along the thinking of Froebel • Drew on the work of Rousseau and Pestalozzi • Sought to teach skills not by repetition but by developing exercises that prepare learners • Worked with children who were considered 'ineducable' • Teacher is decentered; teacher is the 'keeper' of the environment; as learners are active with a task, teachers observe and interview from the periphery (parallels with Dewey) • Participants learn to take responsibility for learning 15 • The child works on self-selected tasks of interest in which the teacher functions as a programmer and a protector of the learning process. (Orem, 1971, p. 40) • The curriculum focuses on the mastery of ones' self and the environment. (Montessori, 1946/1963, p. 87) Vygotsky (1896- 1934) • Social interaction plays a essential role in cognitive development • Zone of proximal development (ZPD) • Consciousness as the end product of socialization • Work is complimentary to the work of Bandura (social learning and situated learning theory); often contrasted with Bruner and Piaget • Did not have an understanding of Piaget's constructivist theory as he died before it was fully developed • Knowledge within a discipline is important, but solving problems that encourage students to go beyond their current skill and knowledge level further development of higher functions beyond the bounds of that discipline. By implication, new knowledge can be built • Vygotsky's ideas influenced a "social constructivist" approach to education. • Proposed that social interaction profoundly influences cognitive development Piaget (1896- 1980) • Introduced the term 'constructivism' in psychology • Focused on knowing rather than being • Cognitive development is facilitated by providing activities or situations that engage learners and require adaptation • Avoid learners performing tasks beyond their current cognitive capabilities • Use teaching materials that actively involve students and present challenges • Piaget did write on Vygotsky's ideas in relation to his own (published in the early 1960's by MIT press as an addition to the 2 n d printing of Vygotsky's 'Thought and Language'). • Difficult author as he never produced a concise comprehensive summary of his ideas; rather they are scattered in many different writings Carl Rogers (1902- 1987) • Drew on the work of Dewey (one of his teachers was a student of Dewey) • Focus was on relationship, "the facilitation of significant learning rests upon certain attitudinal qualities that exist in the personal relationship [sic] between facilitator and learner" (Kirschenbaum & Henderson, 1990, p. 305) • Began to explore the notion of 'student-centered teaching' in his work Client-Centered Therapy (Rogers, 1951, p. 391) • "We cannot teach another person directly; we can only facilitate his learning.. .the educational situation which most effectively promotes significant learning is one in which 1) threat to self of the learner is reduced to a minimum, and 2) differentiated perception of the field of experience is facilitated" (Barrett-Lennard, 1998, p. 184) 16 • Cautioned against underestimating the contribution of 'teaching' and felt there was a role for information transmission Bruner(1915- ) • Learning is an active, social process • New ideas/concepts are constructed based on current/past knowledge • Instructor and student should engage in active dialogue • Curriculum organized in a spiral manner, so student builds on what they have already learned • Work is often linked to child development research especially Piaget • Bruner disagreed with this viewpoint of Piaget (that children were only able to accept information at specific levels of development and at no time before) explaining that, i f materials are presented in an appropriate manner, they can be taught at any age. Readiness was something that should be taught while providing opportunities for learning, not waited for (Bruner, 1983, p. 135) von Glasersfeld (1917- ) • Modified Piaget's constructivism by adding 'radical' to cover the further developments by Heinz von Foerster, Humberto Maturana and himself • Basic principles of radical constructivism are the following: o Knowledge is not passively received either through the senses or by way of communication, but it is actively built up by the cognizing subject and o The function of cognition is adaptive and serves the subject's organization of the experiential world, not the discovery of an objective ontological reality (von Glasersfeld, 1988, p. 83) • Principles are built on the ideas of Piaget • He refers to his ideas as "postepistemological" because his radical constructivism posits a different relationship between knowledge and the external world than does traditional epistemology (von Glasersfeld, 1993, p. 24) As can be seen by the precis many of the beliefs of earlier philosophers and theorists - ideas such as learning should be a series of connections, education is a social process with school a form of community, learning rests on the relationship between learner and teacher, teachers lead learning through discovery, connecting the past experiences with the new experiences, solving authentic real life problems encourage learners to go beyond their current skill and knowledge, learning is an active process -resonate and are found in present-day discussions of constructivism. Constructivism can 17 find its roots in the works of Vico, Hegel, Rousseau, Kant, Montessori, Froebel, James, Dewey, Rogers and Piaget. Thus, current contemporary constructivism is, for the most part, a collection of ideas that can trace its pedigree well back into the history of educational theory. Fundamentally, the difference between early theorists and current contemporary constructivism is the attempt at being not only multi-faceted but also multi-dimensional within the context that it is being used. This complexity gives it not only its strength but also its weakness, as it then becomes difficult to implement this complexity into the traditional Western classroom and educational systems, as they currently exist. It is also important to remember that each of these contributions were originating from different perspectives (Reusser, 2002), although many currently are applied today in the academy and in e-learning. Constructivism can also be considered a product of critical theory, with roots in postmodernist thinking. Kincheloe and McLaren (1994) note that many academics who came of age in the 1960s and were excited by critical theory's basic exploration of experience, and how it was shaped by the social interaction of individuals within communities, began to look at their own disciplines as representative of the "...discourses and power relations of the social and historical contexts that produced them. In trying to find a more egalitarian and open 'social order,' these academics began to explore new ways of thinking within the social sciences" (p. 139). In addition, critical theory created within academics and researchers a strong and dynamic condemnation of the positivist conception of science and instrumental rationality (Schwandt, 1990). 18 Kincheloe and McLaren go on to suggest that in "...postmodernist social theory" academics and researchers began to discard the belief that there is only one truth and to follow the idea that "...reality is socially constructed or semiotically posited" (p. 143). From then on, a standard and understood agreement within the scientific community as to "...what should constitute and guide scientific practice and argumentative consistency [became] an intellectual target for epistemological uncertainty" (Kincheloe & McLaren, 1994, p. 143). What this meant for those working with learning theory was recognition that 'one size did not fit all, ' that there were in fact alternate realities within each individual. This idea has had a dramatic impact on teaching and learning. In trying to explain the issues of postmodernism and constructivism, Wilson, Osman, Jouchoux and Teslow (1997) suggest that "...There may be some confusion as to how postmodernism is different from constructivism - certainly the more common term found in the ID literature. I confess to some confusion myself, and to occasionally mixing up the two terms" (p. 7). Wilson (1997) continues, proposing that ".. . it helps to clarify the issue to think of postmodernism as an underlying philosophy about the world, and constructivism as a very general theory of cognition, suggesting how the mind works and how we know things. The roots of many constructivist beliefs about cognition are traceable to postmodern philosophies which depart from the rationalist, objectivist, and technocratic tendencies of 'modern' society" (p. 8). But even having said this he does offer a caveat. "In truth, not all constructivists are postmodern in their orientation. In psychology, constructivism originally reflected the thinking of people like Piaget and Vygotsky, who were basically modern in orientation.. .It is possible to have a constructivist view of cognition while still retaining a fairly traditional, modern view of 19 science, method, and technology" (p. 8). In examining the issues raised by Wilson, I find myself agreeing with his views regarding postmodern thinking and constructivism. Fundamentally it is possible to be a constructivist and retain a modern world-view. Constructivism as a Continuum Beyond the discourse on where and how constructivism emerged lies another discourse, framed around defining constructivism. For many authors participating in this discourse, "Constructivism is not a unitary theoretical position; rather, it is a continuum" (Doolittle, 1999, p. 1). And it is a continuum that has various junctions along the way. For some, constructivism can be divided into two camps: cognitive constructivism and social constructivism. Others (McMahon, 1997; Matthews, 1992; Murphy, 1997a) divide it into three camps, cognitive, social and radical constructivism. Still others such as Steffe and Gale (1995) note six different constructivist paradigms. These include "social constructivism; radical constructivism; social constructionism; information-processing constructivism; cybernetic systems; and sociocultural approaches to mediated action" (Hausfather, 2001, p. 15). What is fair to say is that one aspect of constructivism that challenges easy understanding is the fact that there is not just one constructivist theory, but a multiplicity of theories (Fosnot, 1996; Phillips, 1995; Prawat, 1996). In order to effectively evaluate constructivism and to frame the debate to some extent, Glasersfeld (1990; 1984) proposed three essential epistemological tenets of constructivism, to which a fourth has been added in light of writings in the late 20 t h century. I believe these tenets continue to hold true into the early 21 s t Century. 20 1. Knowledge is not passively accumulated, but rather is the result of active cognizing by the individual. 2. Cognition is an adaptive process that functions to make an individual's behavior more viable given a particular environment. 3. Cognition organizes and makes sense of one's experience, and is not a process to render an accurate representation of reality. 4. Knowing has roots in both biological/neurological construction, and social, cultural, and language-based interactions (Doolittle, 1999, p. 1). While Fosnot (1996) suggests that "Constructivism is a theory about learning, not a description of teaching..." and that no " 'cookbook teaching style' or pat set of instructional techniques can be abstracted from the theory and proposed as a constructivist approach to teaching" (p. 29), she does offer some general characteristics or main principles to keep in mind, as educational practices are realigned. Learning is not the result of development, learning is development. It requires invention and self-organization on the part of the learner. Thus teachers need to allow learners to raise their own questions, generate their own hypotheses and models as possibilities, and test them for viability. Disequilibrium facilitates learning. Errors need to be perceived as a result of learners' conceptions and therefore not minimized or avoided. Challenges, open-ended investigations in realistic, meaningful contexts need to be offered, thus allowing learners to explore and generate many possibilities, both affirming and contradictory. Contradictions, in particular, need to be illuminated, explored and discussed. Reflective abstraction is the driving force of learning. As meaning-makers, humans seek to organize and generalize across experiences in a representational form. Allowing reflection time through journal writing, representation in multi-symbolic form, and/or discussion of connections across experiences or strategies may facilitate reflective abstraction. 21 Dialogue within a community engenders further thinking. The classroom needs to be seen as a "community of discourse engaged in activity, reflection, and conversation" (Fosnot, 1989). The learners (rather than the teacher) are responsible for defending, proving, justifying, and communicating their ideas to the classroom community. Ideas are accepted as truth only insofar as they make sense to the community and thus rise to the level of "taken-as-shared." Learning proceeds toward the development of structures. As learners struggle to make meaning, progressive structural shifts in perspective are constructed - in a sense, "big ideas" (Schifter & Fosnot, 1993). These "big ideas" are learner-constructed, central organizing principles that can be generated across experiences and that often require the undoing or reorganizing of earlier conceptions. This process continues throughout development (Fosnot, 1996, p. 29). Cognitive Constructivism Cognitive constructivism represents one end of the constructivist continuum, and is typically associated with information processing and its reliance on the component processes of cognition. The roots of cognitive constructivism are found primarily in the first two of the essential epistemological tenets noted by Glasersfeld (Doolittle, 1999, p. !)• Saxe (1991) suggests that cognitive constructivism is a "dialectic process in which the subject resolves perturbations in the coherence of his or her structuring activities by coordinating and constructing new, more adequate cognitive structures" (p. 45). 22 Cognitive constructivism is the closest to the traditional models of cognitive learning offered. The student continues to learn from the teacher as well as from facts and experiences, but the learning takes on a dual role, that of public and private components (Henriques, 1997). Meaning is made and knowledge constructed when learners interact with the physical world (public) and then reflect and make sense of their interactions (private). It is the beginning of the teacher and learner assuming new roles. Social Constructivism The lack of explicit approaches in cognitive constructivism is where social constructivism may offer opportunities. Pioneered by theorists such as Vygotsky, (1978), this paradigm argues for the importance of culture and context in forming understanding. Learning is not a purely internal process, nor is it a passive shaping of behaviours. Vygotsky favoured a concept of "learning as a social construct which is mediated by language via social discourse" (McMahon, 1997). Social constructivism, according to Murphy (1997b), in contrast to cognitive developmental constructivism, identifies different effects in the construction of knowledge. Hodson and Hodson (1998) suggest that "...individual development is shaped as much by cultural as by biological factors, as much by social factors as by individual effort" (p. 37). In other words, knowledge is not merely constructed alone by the individual learner; rather, it is co-constructed through social interaction (Murphy, 1997b). Social constructivist interpretation is based on the belief that knowledge is produced by the interaction of people within a community or society (Airasian & Walsh, 23 1997; McCarthy & Raphael, 1992). At its most intense, social constructivists believe that teaching and learning are in and of themselves, public activities and only after a new idea or theory is 'negotiated' and peer-consensus reached, is it considered legitimate (Prawat & Floden, 1994; Henriques, 1997). A l l sorts of groups and gatherings can craft meaning; political, educational, and economic as well as scientific. Learning occurs when there is internalization of ideas between folks in these groups. Knowledge is assembled through individual exchanges within the sociocultural environment, and the ideas held by the group are considered valid as long as consensus exists (Brooks & Brooks, 1993; McCarthy & Raphael, 1992, Henriques, 1997). Radical Constructivism If cognitive constructivism is at one end of the continuum, then radical constructivism represents the other end of the continuum. Doolittle (1999) notes that radical constructivism adopts completely the first three epistemological tenets of Glasersfeld. "In addition, there is a current movement within radical constructivism to more fully accept the fourth epistemological tenet, thus recognizing social interactions as a source of knowledge" (Larochelle, Bednarz, & Garrison, 1998, p. 5). Jonassen (1991a) posits that the underlying notions of constructivism are deeply unlike those of objectivism. He suggests that radical constructivists (Goodman, 1984; Glasersfeld, 1984; Watzlawick, 1984) accept that there is no indisputable world, no objective reality that is autonomous of human mental activity. In Goodman's analysis, our individual world is fashioned by the mind, so no one world is any more existent than any other. "...There is no single reality or any objective entity that can be described in 24 any objective way; rather, the real world is a product of the mind that constructs that world" (Jonassen, 1991a, p. 29). Jonassen suggests that a less radical form of constructivism embraces the opinion that "...the mind is instrumental and essential in interpreting events, objects, and perspectives on the real world, and that those interpretations comprise a knowledge base that is personal and individualistic. The mind filters input from the world in making those interpretations" (Jonassen, 1991b, p. 10). Radical constructivism also impacts on the role of the teacher and the learner. In a radical constructivist learning situation, each individual's viable understanding of the learning is equally valid; teachers focus on divergent outcomes and consensus among groups of students is not sought. Comparisons between what the individual has come to understand and what is the prevailing belief within the scientific community is not fostered (Henriques, 1997). From Learning Theory to Instructional Theory According to Snelbecker (1985, p. 129), in 1959 the editors of the Harvard Educational Review published a set of papers from a symposium, asking "Can the laws of learning be applied in the classroom?" Over the next two decades, educators took the lead by analyzing the educational context first, then looking to psychology - in particular theories of learning - to see i f there were implications for classroom practice. While learning theories are considered descriptive, instructional theories are described as prescriptive. Gagne and Dick (1983) described instructional theories as follows: Theories of instruction attempt to relate instructional events to learning processes 25 and learning outcomes.... They attempt to identify conditions of instruction which will optimize knowledge acquisition, retention, and transfer... Instructional theories attend to the arrangement of features in the learner's environment, to intentionally promote learning, (p. 264) Bruner (1966) noted that instructional theories should have a high degree of generality. An instructional theory should specify: • The ways in which a body of knowledge [content] should be structured so that it can be followed by the learner; • The most effective sequencing of content; • The pacing of the content; • Those experiences/activities which are most able to predispose learners to engage with content. (Hartley & Davies, 1978, p. 41) In accordance with these criteria, there are numerous instructional theories used in developing instructional and teaching events. They make practical and applicable the psychological theories of learning, by providing prescriptions for classroom and/or distance instruction. They range from those based in behavioural learning theory, to those based in constructivist learning theory. Commonly used instructional theories include mastery learning, programmed instruction, cooperative learning, problem-based learning, authentic learning, and collaborative learning. 26 Constructivism and the Instructional Design Framework Instructional design has as its basis for practice four theoretical underpinnings: general systems theory, communication theory, learning theory, and instructional theory. Two of these theory bases - learning theory and instructional theory - have application to this study through instructional design (McGriff, 2001). Gagne and Dick (1983) describe the key features of instructional design as an attempt to relate specified events of instruction to learning processes and learning outcomes. Instructional design models are: • framed by the knowledge base of learning research and theory; • instructionally focused and prescriptive in the sense that they attempt to identify conditions of instruction which will optimize learning, retention, and learning transfer; • expected to provide, at a minimum, rational description of causal relationships between procedures used to teach and their behavioral outcomes, preferably enhanced human performance, (p. 267) From the first advent of the use of constructivist principles or ideals in instructional design, an inherent conflict became apparent. How does the instructional designer develop constructivist-focused learning via instructional design? No matter how good the intentions, the basic tenets of constructivism seem to require that "because 'the constructivist model is descriptive, not prescriptive' (Airasian & Walsh, 1997, p. 444), it is difficult to define instructional strategies that always support constructivism" (Miller, 2002). Miller (2002) goes on to suggest "that any instructional strategy has the potential 27 for supporting student construction of knowledge... [including a] least-likely method—the lecture—as an example of how any instructional strategy can potentially support student construction of knowledge" (p. 3). As Dalgarno (2001) indicates, the focus of instruction and teaching, within a constructivist milieu, is on guidance toward knowledge construction - rather more difficult to design for than knowledge transmission models. Airasian and Walsh (1997) warn and I agree that is important to not see constructivism as anything more than "an epistemology of learning" as opposed to "a well thought out and manageable instructional approach" at this point in the study of constructivism. Basically, there is currently no "cook book version" of how to do constructivism. "Instead there are suggestions for methods and strategies that are more likely than others to foster the desired traits in students such as knowledge construction, intrinsic motivation, independent learning and responsibility" (Dobozy, 1999, 1|33). Wilson (1997) adds to the debate by advising that "...The field of instructional design, evolving from behavioral psychology, systems technology, and management theory, sees the world through the 'scientific' lens, whereas postmodernists tend to see things through a critical, humanities type of lens. The goal of an artist or critic is not so much to explain, predict, and control, but to create, appreciate and interpret meanings" (p. 4). I believe that this movement to postmodern ID is the future of instructional design and is one of the driving forces behind the attraction to constructivism in creating online learning strategies. Although recognizing the potential conflict of constructivist theory and planned or designed instructional events, instructional designers continue to work toward the 28 incorporation of many of the philosophical tenets of constructivism into instructional design frameworks, in order to provide creators of instruction with a set of very broad rules and strategies. Broad rules and strategies might aid in the creation of a constructivist learning environment, a useable model and possibly lead to a theory of constructivist instruction. According to Mergel (1998), The shift of instructional design from behaviorism to cognitivism was not as dramatic as the move into constructivism appears to be, since behaviorism and cognitivism are both objective in nature. Behaviorism and cognitivism both support the practice of analyzing a task and breaking it down into manageable chunks, establishing objectives, and measuring performance based on those objectives. Constructivism, on the other hand, promotes a more open-ended learning experience where the methods and results of learning are not easily measured and may not be the same for each learner, (p. 20-21) This attempt to merge what appear on the surface to be two different world-views goes to the very core of the dilemma faced by instructional designers today. While behaviorism and constructivism are very divergent theoretical perspectives, cognitivism shares some affinity with constructivism. Mergel (1998) goes on to suggest that "Despite these similarities between cognitivism and constructivism, the objective side of cognitivism supported the use of models to be used in the systems approach of instructional design. Constructivism is not compatible with the present systems approach to instructional design" (p. 21). As Jonassen explains: "The conundrum that 29 constructivism poses for instructional designers, however, is that i f each individual is responsible for knowledge construction, how we as designers can determine and insure a common set of outcomes for learning, as we have been taught to do?" (Jonasson, n.d., 1fl0). He notes that "Although we believe that constructivism is not a prescriptive theory of instruction, it should be possible to provide more explicit guidelines on how to design learning environments that foster constructivist learning" (Jonasson, n.d., T]10). Jonassen (n.d.) points out that the difference between constructivist and objectivist (behavioral and cognitive) instructional design is that objectivist design has a predetermined outcome and intervenes in the learning process to map a pre-determiried concept of reality into the learner's mind. Constructivist design maintains that because learning outcomes are not always predictable, instruction should foster, not control, learning. With this in mind. Jonassen (n.d.) looked at the commonalties among constructivist approaches to learning to suggest a "model" for designing constructivist learning environments. He believes that "...a constructivist design process should be concerned with designing environments which support the construction of knowledge..." (Mergel, 1998, p. 21). (See Table 2.) Important in this thinking is the emphasis on internal and social negotiation, both internally and externally; the exploration of real world contexts and finally the teacher modelling leading to facilitation or guiding rather than transmission or lecture allowing for co-learning versus teacher always the expert. The technological advances of the 1980s and 1990s have enabled designers to move toward a more constructivist approach to the design of instruction through the use 30 of hypertext and hypermedia as they allow for branched designs rather than linear formats of instruction. Not without concerns, Jonassen and McAlleese (n.d.) note that Table 2 Jonassen's Suggested "Model" for Designing CLE Is based on internal negotiation Is based on social negotiation Is facilitated by exploration of real world environments and intervention of new environments Results in mental models and provides meaningful, authentic contexts for learning and using the constructed knowledge Requires an understanding of its own thinking process and problem solving methods Modelled for learners by skilled performers but not necessarily expert performers requires collaboration among learners and with the teacher Provides an intellectual toolkit to facilitate an internal negotiation necessary for building mental models A process of articulating mental models, using those models to explain, predict, and infer, and reflecting on their utility (Piaget's accommodation, Norman and Rumelhart's tuning and . restructuring.) A process of sharing a reality with others using the same or similar processes to those used in internal negotiation Processes that are regulated by each individual's intentions, needs, and/or expectations Should be supported by case-based problems which have been derived from and situated in the real world with all of its uncertainty and complexity and based on authentic realife practice Problems in one context are different from problems in other contexts The teacher is more of a coach or mentor than a purveyor of knowledge Source: Jonasson, D.H. (n.d.). Thinking technology: Toward a constructivist design model. Retrieved February 12, 2005 from http://web.archive.Org/web/l 9990427173629/http://ouray.cudenver.edu/~slsanfor/cnstdm.txt each phase of knowledge acquisition requires different types of learning and that initial knowledge acquisition is perhaps best served by classical instruction with predetermined 31 learning outcomes, sequenced instructional interaction while the more advanced second phase of knowledge acquisition is more suited to a constructivist environment. I believe this 'developmental' notion of the role of constructivism in creating instruction is becoming more prevalent among those who practice instructional design. In my own work, I have begun to see movement toward the recognition that constructivism is many-layered. This has implications, I believe for not only the stage of the learner (age across the curriculum, preparedness for self- and group-exploration) as well as the content or context (beginning principles vs. advanced understanding). It is also becoming a question with respect to the idea that constructivism's 'internal and social negotiation' does not negate structure or foundation upon which to build. Bull, Montgomery and Kimball (1999) propose, and as an instructional designer I consider accurate in current ID practice, a number of components they feel inform constructivist instructional design: When designing instruction one should look at the intent of the instruction rather than the behavior which it might engender. Support learner goals to promote motivation. There are many ways to do things when diverse learners are involved. Define the content in multiple ways with cases, patterns, and stories. Provide rich experiences, which may encompass the desired outcomes. Teach learning-to-learn skills so that students can construct their own learning. Appreciate interdisciplinary goals and methods. Give priority to problem solving and meaning constructing goals. Be sensitive to the value implications of the instructional decisions which are made. Use the end users, students, and teachers 32 as part of the design team so that their voices will be heard. Consider that there are multiple stages of expertise not just novice and expert and that each level needs somewhat different instruction if they are to learn effectively and efficiently. This would lead to use of a layers of need model for instructional development. Some learners need to be on the fast track and others will only be successful i f they have more time. (TJ77) When operating within the framework of constructivist instructional design, it is more suitable to talk about the design of learning environments than about the design of instructional events. The work of Wilson and Cole (1991), Honebein (1996), Cunningham, Duffy, and Knuth (1993) and Jonassen (1999) all attempt to create frameworks for what have been called constructivist learning environments [see also Murphy, 1997b, Jonassen, 1991b, and Moallem, 2001]. A summary of key elements of those environments, based on the work of Jonassen (1991b), Murphy (1997b), Wilson and Cole (1991), and Honebein (1996) includes: • Learning is rooted in a rich environment where authentic and academic contexts for learning tasks are accounted for. Real world environments ensure that the learning is relevant and focused on real-world problem solving, helping learners to interpret the multiple perspectives of the world. Learning environments stress conceptual interrelatedness, providing multiple representations or perspectives on the content. • Provisions for learner control are incorporated and learning is internally controlled and mediated by the learner. 33 • Errors are used as a mechanism to provide feedback on learners', understanding, and evaluation serves as a tool for self-analysis. • The environment allows for a student-centred learning process whereby learners play an important role in setting the goals for learning. Instructional goals and objectives are negotiated, not imposed. Learners are provided with experience in order to best utilize the knowledge construction process. • Learning is embedded in social experience. There is a provision for collaboration, and metacognitive and reflexive activities are encouraged. • Instructors are coaches and analyzers of the strategies used to solve real-world problems. They provide the learner with opportunities to experience and appreciate multiple perspectives and multiple modes of representation. There is little question that historically dominant instructional design approaches seem to be based on theories of learning that have passed out of favour (Ally, 2004). Traditional instructional design models are unable to make a comfortable fit with newer theories such as constructivism. Reigeluth (1997, p. 45) outlines a major shift in the social and intellectual context that is influencing instructional design. He maintains that there is a shift from Industrial Age to Information Age thinking. One of the key markers of this shift that is seriously influencing instructional design, as conceptualized and practiced, is the move from standardization to customization: designers are increasingly attempting to make possible a unique learning experience for each learner, rather than trying to produce a single, clearly-defined outcome for all learners. The combined powers 34 of new communication and computer technologies are aspects of the context which are driving the shift in approach. As for Wilson (1997), he continues his exploration of post-modern instructional design, and, in asking what post-modern instructional design would look like, offers a description, (see Table 3.) Table 3 Wilson's Postmodern Instructional Design (ID) in Traditional Elements General ID Approach Stretch the rules Put principles above procedures Put learners and teachers above principles Needs Assessment Resist temptation to be driven by easily measured and manipulated content Ask who determines what is an instructional need Ask whose needs are being served, and whose neglected Goals, Objectives, Task Al low learning goals to emerge during instruction Analysis Use objectives as heuristics to guide design Define content in various ways, and present through cases, stories, and patterns Develop Instructional Support learners in pursuing their own goals Strategies Design learning environments, not instructional events Consider instructional strategies that provide multiple perspectives and encourage the learners to exercise responsibility Select Media Include both media literacy and media biases for consideration Provide opportunities for learners to use media in their pursuit of understanding Evaluate Instruction Critique and discuss products and performances grounded in authentic contexts Use, where possible, informal assessments within the learning environment Source: Wilson, B. G. (1997). The postmodern paradigm. In C. R. Dills & A. A . Romiszowski (Eds.), Instructional development paradigms. Englewood Cliffs NJ: Educational Technology Publications, in press (to be published in March 1997). 35 In using the traditional terms of instructional design, he suggests some interesting additions, a 'postmodern twist' as he put it, to incorporate constructivism into traditional instructional design (p. 9). Goldman and Torrisi-Steele (2002) note that when using constructivism, the learner, rather than the teacher, becomes the focus of the learning environment. From a constructivist perspective, the focus of instructional design shifts from being goal-orientated, strictly structured and ordered knowledge transmission, to a process focused on reconceptualization of knowledge acquisition to ensure active exploration by the learner (Brown, 1997). This refocus from teacher to learner has profound effects. The teacher's role changes to a manager of knowledge production, a facilitator who provides advice in exploration, a guide, a helper and an assistant (Prosser & Trigwell, 1999; Marton, Hounsell, & Entwistle, 1997). This role of guide and facilitator is often argued against as 'not teaching'. Jonnassen (1991a) suggests that the "...greatest misinterpretation of constructivism is that it...results in academic chaos. If all learners construct their own meaning from information, how can we share enough knowledge to even communicate?" (p. 31). He argues that we do ".. .share enough to communicate, to argue, to hypothesize... [therefore] if meaning is negotiated, why shouldn't we also negotiate the goals of learning or use the negotiation process in the form of augumentation, as the evidence of learning?" (Jonnassen, 1991a, p. 32). This view is completely in keeping with my own experiences as an online teacher/guide/facilitator. The ability to negotiate learning within a constructivist environment is underrated and I would contend is under supported in many online learning situations. 36 Developments in Online Learning As Ally (2004) indicates, "increasingly organizations are adopting online learning as the main delivery method to train employees (Simmons, 2002). At the same time, educational institutions are moving toward the use of the Internet for delivery, both on campus and at a distance" (p. 5). Developments in technology have permitted distance education to reach beyond its origins, to new groups of learners, new organizational partnerships, and new challenges and possibilities in the design of instruction and Web-based learning environments. Palloff and Pratt (1999) note that information technology has been changing the face of distance education over the past decade (p. 3). Traditionally a correspondence model, with the advent of audio conferencing and to a lesser extent video conferencing,, distance was bridged for both learners and instructors. Now with the advent of asynchronous group communication tools and platforms, distance has been bridged among learners themselves and among teachers and learners, in that all can be engaged in continuous and regular communication, in their own time and space, and distance education has been remolded into online learning. Gabriel (2004) points to the growing demand for distance education and training, encouraged by "The new knowledge economy and the emergence of affordable technologies... with the relatively recent history of Web-based course delivery, new understandings of the potential role of institutions, instructors, and learners have begun to emerge in the literature" (p. 54-55). Palloff and Pratt (1999) provide an outline of some of the changes that are occurring because of advances in technologies of learning. 37 Courses and degree programs are being offered over the Internet. Virtual universities are being constructed, such as the Western Governors' University and the California Virtual University where students can apply for admission, register for courses, purchase books, and attend classes without ever visiting a physical place called a campus, (p. 4) Since Palloff and Pratt's text, many advances have taken place in computer-mediated or online learning. Here in Canada, many traditional universities have taken advantage of new technologies to expand their reach to students beyond their geographic boundaries, by offering undergraduate and/or graduate degree, diploma, and certificate programs online. Virtual universities are being founded - universities with no physical campus and which students never see. University alliances on a national and an international scale are being formed. Professional development associations and training organizations have embraced the use of online technologies to offer just-in-time training and professional growth to those in the work force. "Technological advances in delivering courses, workshops, and seminars have demanded a parallel development of effective teaching methods and learning strategies" (Gabriel, 2004, p. 55). Ally (2004) notes that it is difficult to develop a generic definition for online learning. "Terms that are currently used include e-learning, Internet learning, distributed learning, networked learning, tele-learning, virtual learning, computer-assisted learning, Web-based learning, and distance learning" (p. 4). The University of Western Australia (2001) provides what I consider to be one of the better definitions of online learning in terms of learning environment. 38 An online learning environment is one that goes beyond replication of learning events that have traditionally occurred in the classroom and are now made available through the Internet. It provides for different ways of learning and the construction of a potentially richer environment that provides for fresh approaches to learning, caters to different learning styles, as well as allowing for greater diversification in learning and greater access to learning. (^ ] 3) Sloan-C View (2003) suggests that online learning can also "include formal and informal learning, [and] the terms may be used for hybrid or blended courses in which components of instruction are face-to-face and online. (^  4) Online learning is carried out in asynchronous learning mode, for the most part. Asynchronous communication, capable of occurring at any time and at irregular intervals, means that learners and instructors can communicate online without a pattern of interaction (Palloff & Pratt, 1999). In a document written online by the Australian Flexible Learning Framework for the National Vocational Education and Training System 2000-2004 (2003), it is stated that: "Asynchronous interaction involves the parties communicating over elapsed time, not real time, usually in a typewritten format. As well as straightforward discussions, asynchronous interaction could also include group project activity, assessments, surveys, votes, etc. These activities may be completely open-ended or may be constrained with a defined start or end time" fl[ 9). 39 Integrating Constructivism, Instructional Design, and Online Learning The literature on learning theory and specifically on constructivist learning theory, on instructional theory, on instructional design models and frameworks, and on online learning can be brought together, in that learning and instructional theories inform instructional design decisions, which in turn impact on the design of constructivist, online learning environments. Ally (2004) delineates constructivist learning and the implications for the design of online learning environments. 1. Learning must be active. Keeping learners active doing meaningful activities results in high-level processing, which facilitates the creation of personalized meaning. 2. Learners should construct their own knowledge rather than accept that which may be provided by an instructor. Knowledge construction is facilitated by good, interactive online instruction, which requires students to take the initiative to learn and to interact with other students and the instructor, and because the learning agenda is controlled by the student (Murphy & Cifuentes, 2001). 3. Collaborative and cooperative learning should be encouraged to facilitate constructivist learning (Hooper & Hannafin, 1991; Johnson & Johnson, 1996; Palloff & Pratt, 1999). Working with other learners gives learners real-life experience of working in a group and allows them to use their metacognitive skills. Learners are also able to use the strengths of other learners, and to learn from others. 40 4. Learners should be given control of the learning process. There should be a form of guided discovery where learners are allowed to make decisions on learning goals, but with guidance from the instructor. 5. Learners should be provided with time and opportunity to reflect. When learning online, students need the time to reflect on and to internalize the information. Embedded questions can be used to encourage learners to reflect on and to process the information in a relevant and meaningful way. 6. Learning should be made meaningful for the learners. Assignments and projects should allow learners to choose meaningful activities to help them apply and personalize the information. 7. Learning should be interactive to promote higher-level learning and social presence, and to help develop personal meaning. Interaction is also critical to creating a sense of presence and a sense of community for online learners, and to promoting transformational learning. In the transformational process, learners interact with the content, with the other learners, and with the instructor to test and confirm ideas and to apply their new knowledge (p. 18-20). There is as yet no real agreement on what constitutes constructivist learning, or on how it manifests itself in teaching/learning situations. There are suggestions for applying constructivist thinking to instructional situations, both face-to-face and online, many of which are founded in practice. But as Young (2003) notes "the formal description of constructivism deviates from the actual practice in technology-enhanced [online] 41 environments" (p. 62). In order to maintain the integrity of constructivist, computer-based learning, the environment must remain flexible, customized, and dynamic in its structure in order to facilitate decision-making by students (Takala, Hank, & Rammos, 2001). Summary This literature review has covered a number of different aspects of the evolution of constructivism, constructivism and instructional design, instructional theory and current developments in online learning. As can be seen from many of the writers I chose to examine, from its original roots, constructivism had entered into both the traditional classroom (although slower than many constructivists would have liked and faster then many traditionalists felt was necessary) and the online classroom, leaving many teachers and researchers to find the online classroom an ideal arena for the testing of many of the assumptions of constructivism theorists. Like all new enthusiasts, early adopters of constructivism in online learning took what they knew and applied it to this new delivery model, which seemed perfect for the tenets of constructivism. And like all new enthusiasts, as time went on, ideas were modified, adjusted, thrown out or adopted as having merit for the improvement of teaching and learning pedagogy. Instructional designers and instructors, like myself, were left with many and varied suggestions and ideas of practice, some still at the theory stage and many in various forms of practical advancement, and documented in conference proceedings and academic journals. This largesse became the quandary. Much of the literature revealed 42 not only the strong positives about constructivism but also some of its flaws. This then posited the question, that after many years of discussion and debate amongst academics and those practicing ID and online learning, could a consensus possibly bring the discussion closer to what actually constitutes constructivism and could that discussion affect how does an instructor or designer grapples with the subtle nature of teaching and learning in a constructivist environment. Especially when constructivism, at least on the surface, asks the teacher and the student to fundamentally change the way they think about teaching and learning, and in turn, impacts on how students are assessed and courses/programs evaluated. 43 CHAPTER 3 DESIGN A N D M E T H O D O L O G Y This chapter reviews the literature on conducting online research and describes the methodology selected to conduct this study - namely the Nominal Group Technique (NGT). It presents the decision-making process of selecting NGT, the design, and process by which the plan was implemented during the study. Conducting Research Online According to Anderson and Kanuka (2003), online research is "more than a new set of research techniques" (p. 5): The e-Researcher is both a participant and a researcher of the environment in which the research occurs. E-Research spans temporal distance... [and] research applications can be customized to take advantage of either synchronous or asynchronous formats - or both. E-Research ... is concerned with both the application and adoption of tools from the real world, and the invention, refinement, and calibration of a new genre of tools, (p. 5-7) The Internet provides the potential for broad expansion of research efforts in education, and, as noted by Kraut, Olson, Banaji, Bruckman, Cohen, and Couper (2003), online research is "changing the way [researchers] collaborate, collect data, and disseminate their results" (p. 105). These authors, citing the works of others, delineate opportunities provided through online research, as follows: 44 • It lowers the costs of collecting data, allowing researchers to run online experiments with potentially thousands of participants (Nosek, Banaji, & Greenwald, 2002). • It has the potential to be conducted with minimal intervention on the part of the researcher, and over a much shorter time frame than traditional research [especially in the case of survey research] (Nosek, Banaji, & Greenwald, 2002; Orlikowski, 2000). • New social phenomena can be explored, such as behaviour in virtual groups, distributed or online discussion groups, and online collaboration (Galegher, Sproull, and Kiesler, 1998; Hinds and Kiesler, 2002). Traditional research methodologies have been used in online research and have included survey research, virtual observational research, content analysis, and random assignment experiments. In conducting online empirical research, data collection tools and procedures used include participant observation, structured and semi-structured interviews - in real time using synchronous communication tools and virtually using asynchronous communication tools, focus groups, virtual interviews, consensus studies, and content/document analysis. Many researchers, in moving into online research, carried with them techniques and methodologies which they had used in traditional research settings. While traditional methods and approaches may indeed work in online research, many call for new research methodologies that are able to take into account the properties and exceptional characteristics of new information technologies, arguing that long-45 established research methods are unfitting because they were created for a different environment (Johnston, 1984). Harasim (1991) agrees: "New communication media, particularly] computer-mediated systems such as electronic mail, computer conferencing and bulletin boards ... [enable] new forms of educational interaction to study as well as new tools for conducting such research" (p. 1-2). As Rossman and Wilson (1994) suggest, "Perhaps wisdom lies in being tolerant and shamelessly eclectic in our use of methods" (p. 320). Along with the new areas to explore and research, "research on education, in general, is a discourse, with its own tools and practices.. .through the utilization of digital technologies...the very structure of the discourse of educational research is being altered" (Middleton, 2000, p. 3). From the researcher's perspective, what are some of the advantages and disadvantages of this new arena for research? I have identified a number of possibilities and challenges, which are discussed in detail below. Possibilities • The extension of access to participants for research purposes. • The cost and time saving of using new technologies to support research. • The ease with which large amounts of data can be handled, including the reduction of transcription bias, common in qualitative research. Challenges • Ethical issues and treatment of participants. • Sample bias and problems of generalizability. • Computer skills of both the researcher and the participants. 46 • Contact, recruitment, and retention of participants. Access to participants. The Internet and its users are not just next door to the researcher. They are across the street, across the city, across the country and around the world. What was once almost impossible to do, given time, costs and geography - that is, conduct research on a project that might span a continent - is now possible. The Internet is a "...global system accessing local newsgroups in many countries and in many languages, allowing cross-cultural comparisons of issues" (Coomber, 1997, p. 34). Kraut et al. (2003) indicate that online research has "democratized data collection" (p. 106). Not only does geography play a role in the world of online research, but access to hitherto less accessible groups is also feasible. Hard to reach populations (stay-at-home mothers with small children, shift workers, shut-ins, the disabled) all may be more accessible to online researchers. The possibility of broad access also extends research into areas that might be potentially dangerous or politically sensitive (Mann & Stewart, 2000, p. 18). Cost saving and time saving. This is an area where both qualitative and quantitative researchers maintain that the Internet and online research environments have significant advantages over the face-to-face research environment. Once hardware and software are purchased, the main fees are monthly telephone and ISP (Internet Service Provider) expenses (and these can often be negligible, depending on the host for the research). Online survey research, for example, is the least expensive research methodology. For a given sample size, online surveys can, i f done properly, be executed for less cost than any traditional form of research. In addition, the researcher can benefit 47 from faster turnaround time and higher response rates (Batagelj & Vehovar, 1998; Solomon, 2001). Savings can also include time and travel, and associated expenses, usually undertaken by both researchers and participants. As Mann and Stewart (2000) indicate, this can lead to "...compromise in regard to where interviews are held and with whom" (p. 21). Online research allows for the accomplishment of research that might be outside most conservative resources. "Online research eliminates the cost (and time) barriers presented by travel" (Mann & Stewart, 2000, p. 21). Another area of saving closely related to travel is the rental of a site for the research - the location for the face-to-face interactions. Online research avoids such expenses, as it is conducted in virtual space (Mann & Stewart, 2000, p. 21). Kraut et al. (2003) support this stance on reduced cost, noting that "a primary advantage of the Internet for both survey and experimental research is the low marginal cost for each additional research subject. Unlike traditional laboratory experiments or telephone surveys, where each new subject must be greeted, instructed, and supervised by a person, most online experiments and surveys are automated" (p. 107). Ease of data handling. Another advantage for researchers using the online research environment is the easy handling of the data. As Kraut et al. (2003) indicate, "because people communicate online using text, these conversations are pretranscribed. The use of automated coding and content analysis tools, such as MacWhinney's (2000) C L A N software...further speeds the research process" (p. 106). Transcription costs are minimal, since there is little need for transcription of audiotapes, and transcription bias is 48 eliminated, if conducting interviews or focus groups, since the data are presented in the original words of the participants, in full text. Qualitative research, especially, can require the organization of a great deal of data. Field notes can be easily created and/or coded in a number of ways through the use of software programs such as QSR N6, QSR NUD*IST, Vivo, or Ethnograph (Cresswell, 2003). Transcript analysis is a particularly apt method of data review, given the text-based exactness of online communication. Every contact is carried out and logged online and by the computer. Kraut et al. (2003) indicate that researchers can capture "metrics such as time online, response latencies, changed answers, or backing up, permitting richer analysis across subjects" (p. 107). As Harasim (1991) notes ".. . [t]he use of hypertext to conduct transcript analysis is a promising new research technique." Harasim goes on to caution that "[t]he text-based nature of online interaction provides the researcher with an automatic transcript, which has been unobtrusively gathered. (The very ease of obtaining such a record makes ethical considerations all the more critical)" (p. 6). Ethical concerns and issues. While online research has no greater potential to adversely affect human subjects, the online environment "changes the nature of the risks" (Kraut et al., 2003, p. 108). Issues such as expectations of privacy, anonymity, and informed consent need to be rethought, and possibly new procedures developed for use in online research. For example, it is common practice to assume informed consent if the initial contact is made, and the potential participant returns via email a copy of the consent form with his/her permission entered in the form. But it is difficult, in these circumstances, to gauge how informed the consent is, and whether indeed it has been agreed to by the initial contact. 49 Similarly, public versus private behaviour becomes an ethical issue. Is it ethical to use public Internet-based communication without the knowledge or consent of participants, because it is assumed that this is a record of public behaviour? The current stance is determined on the basis of the expectation of privacy, which may vary from Internet setting to setting. Several researchers felt this issue should be judged 'on a case-by-case basis' when deciding about the status of online communications among individuals on an electronic distribution list or an Internet chat room (Kraut et al., 2003; Bull & McFarlane, 2000). Sampling in online research. "Unlike random digit dialing of telephone numbers, no sampling frame currently exists that provides a random sample of Internet users. Generalizing from Internet samples to the larger population is especially problematic" (Kraut et al., 2003, p. 107). There are in particular two issues that create sampling problems for online research. First of all, Internet users and non-Internet users, and those with computer access at home versus those without computer access at home differ considerably in demographics. Second, self-selection and dropout rates, particularly in large survey studies, skew the sample. A number of surveys into the demographics of Internet users have consistently found that Internet users are more likely to be white, male, first world residents, relatively affluent and relatively well educated in comparison to any other general population (Kehoe & Pitkow, 1996). Add to this issue the self-selection of participants in large survey research, and the representativeness from sample to population is questionable. 50 There was also some concern that the response rates to online surveys were sometimes lower than mailed or telephone survey response rates (Couper, 2001; Fricker & Schonlau, 2002, Kraut et a l , 2003). Computer skills for the researcher and participants. As Mann and Stewart (2000) state, "...running a CMC-based research project requires some degree of technical expertise on the part of the researcher" (p. 26). Of course, how technically proficient a researcher needs to be depends on his/her choice of methods required for his research, and on whether technical support and management is available. The skills to create and analyze an online survey are very different from those skills needed to moderate an online, real-time chat focus group.. These examples of computer proficiencies, while different for the participant, are no less important. Having few technical skills -especially limited typing or literacy - in using computer-based interactive technology can result in a participant in online research feeling discomfited or trivialized. Many participants in online research might consider themselves to be computer literate, in terms of their everyday use of the computer in their professional lives. But computer generated survey tools and other research software have idiosyncrasies, and participants expect, since they are voluntarily participating in many cases, to be able to proceed with little anxiety. Common complaints on the part of participants include tightly timed responses, inability to move backwards through a survey or to change earlier responses, and inability to access the site more than once (Gabriel, 2004). The researcher, in an online setting, needs not only technical skills, but also those human skills found in qualitative research that create a bond with participants (Rezabek, 2000). Perhaps even more interesting is the need for proficiency of expression through 51 the written word. Without body language and facial expressions to rely on and enhance communication, what is needed on the part of a researcher in an online exchange is to be noticed - that is, to extract a reaction from the other members of the online community the participants (Colomb & Simutis, 1996). Making contact, recruiting, and retaining participants. The issues of making contact, recruiting and retaining research participation for the full term of a study are not unlike those of the face-to-face researcher. The research and the researcher fix on the focus of the study and appropriate participants are identified. Research guidelines usually require that initial contact be made to potential participants by email or via an online connection of some sort. If a study is to explore an already created group of learners or participants, then it becomes a matter of getting permission to access the list and make contact with those on the list. If the intention is to work with unknown participants, researchers can target online newsgroups with recruitment messages asking for help (Coomber, 1997). However, even in public newsgroups, such messages are not usually well received unless the researcher has first asked permission of the virtual community (Foster, 1994, p. 96). As Mann and Stewart (2000) note ". . . [t]he easiest way to find someone's email address is to ask them" (p. 26). There are no global directories, no central lists and it is almost impossible to purchase commercial lists. Even when a researcher requests an email address and it is freely given, it can have typographical errors rendering it useless for contact between researcher and participants. Similarly, a researcher might get a workable email address, only to find the owner does not often use it (Foster, 1994, p. 94). 52 Losing access to the participants during a study is a challenge for online researchers. The key to this challenge is to ".. .sustain electronic connection with participants for the whole period of the research" (Mann & Stewart, 2000, p. 30). This problem is exacerbated if a researcher is working with students who have email access as part of their studies, then graduate, leaving them without access until they purchase a private electronic address (Kingsley & Anderson, 1998). The way around this might be to point participants to the free addresses offered by some companies and agencies on the Internet, that allow access via free terminals in public libraries or through Internet cafes. As well, since Kingsley and Anderson's report in 1998, there has been a significant move to personal email accounts, purchased by an individual. This trend may well nullify any future email contact issues, as email addresses become as established as traditional physical addresses (Mann & Stewart, 2000, p. 30). Structured Group Processes - Consensus Building In examining a methodology for the exploration of my research questions, I concluded that the best option for my research was a consensus methodology, and one that focused on the group rather than the individual. Why consensus? Having formulated my research questions, I turned to Linstone and Turoff (1975), who identified specific circumstances in which a researcher, wishing to obtain group consensus, may find other kinds of communication processes (such as a focus group or interviews) too restrictive. These circumstances include: 53 • The research problem does not lend itself to well-defined systematic techniques; it can, however, acquire useful results from subjective judgments on a collective basis. • The research participants will be representative of diverse backgrounds, with respect to experience and expertise, and are geographically dispersed, making frequent group meetings virtually impossible due to time and cost. • The experts may be uncompromising between opinions, to the extent that the communication process must be refereed and/or anonymity assured. • The heterogeneity of the research participants must be preserved to avoid domination by some experts (cited in Anderson & Kanuka, 2003, p. 120-121). According to Innes (1999), consensus building refers to a range of processes used to foster dialogue, clarify areas of agreement and disagreement, and resolve controversial issues (p. 647). My intent was to foster dialogue among participants around the issues of constructivism and online learning, and to conduct my study online. Anderson and Kanuka (2003) suggest a number of advantages to consensus building techniques via traditional delivery and also via the Internet. They maintain that the following can be accomplished through the use of consensus building approaches: • High-quality, informed opinions - consensus groups are usually selected so that participants are "informed, interested and capable of providing high-quality opinions.. .this creates an environment for social cognition that is likely to produce better decisions than those made by individuals." 54 Safety in numbers - consensus groups are "less likely to arrive at or support incorrect answers or ineffective solutions, [since they]...are working with the collective expertise." Authority - group decisions "are more likely to be taken seriously...consensus techniques have been shown to be more reliable and valid than other forms of opinion gathering and synthesis." Controlled process - They "provide a set of procedures that tend to mitigate the negative impacts of group behavior." Support for communication among individuals with polarized views -Although not always able to reach a "unified position, they do create an environment in which polarized views can be democratically expressed and negotiated." Credibility - While some continue to be critical of the techniques, "...various mathematical techniques can be applied at each stage of the process to quantify individual and group opinions [and] feedback...allows participants to judge their opinions in relationship to the larger group...the extent of consensus can be accurately calculated and discussed." Accessibility - most consensus groups have historically used face-to-face meetings, while others have been supported by computer technology and have allowed for "instant computation of the results"; traditional post or courier services have also been used to ".. .allow members to post and defend their reasons without meeting face-to-face. However, the inherent time delay, inconvenience, and cost of postal returns remain problematic." 55 • Time and cost savings - "The interactive capacity of the Net can be adapted to provide instant feedback to participants as to the extent to which their answers are congruent with those of the other panel members." • Equitable time and power sharing - Assigning aliases to those participants participating over the Internet, can "...control domination by any individual in the group [as]...power and status can be kept confidential." • Broad and diverse opinions - "Because participants in Net-based consensus building research can be located anywhere and generally can participate at any time, it is possible to call on participants from nearly any geographical location." (p. 123-124) Today, the transitional phase continues as researchers attempt to model their best practices for good, solid research in online learning environments. Researchers continue to conduct comparative studies, to use experimental and qualitative approaches, to use online and traditional surveys, interviews, focus groups, and consensus building techniques. As Anderson and Kanuka (2003) note: "...successful innovation relies on high-quality research... as e-researchers we have a tremendous opportunity to make a profound difference in the rapid evolution of network-enhanced research" (p. 207-208). The Decision-Making Process In exploring both consensus building and wholly online research, the Nominal Group Technique (NTG) provided a promising pathway. Before making a final decision 56 regarding the methodology, a brief exploration was undertaken of other possible approaches. The Delphi Procedure The Delphi Procedure has been used to not only to obtain consensus but to forecast the future (Neiger, Barnes, Thackeray, & Lindman, 2001; Bramwell & Hykawy, 1999; Gabriel, Ostridge, & Doiron, 2003). It was created by the R A N D Corporation in the USA in the 1950s . (Neiger, Barnes, Thackeray, & Lindman, 2001; Bramwell & Hykawy, 1999; Gabriel, Ostridge, & Doiron, 2003). According to Fish and Busby (1996) and Linstone and Turoff (1975), the Delphi method is a procedure designed to have a panel of knowledgeable persons reach consensus on a particular topic. A Delphi study is usually conducted in three phases. The instrument used is a questionnaire, which is adapted for each phase (Kreber, 2002). Panelists vote on individual items, and justify their positions through each round. Delphi panels are usually quite large, hence discussion and debate are somewhat limited by the size of the group and by the process itself: ranking is the primary activity, while discussion and debate are secondary. Search Conference Methodology Derived from action research (Rockefeller Foundation, 1997), a modified search conference methodology is a technique for participative planning in which a group of 20 to 35 people come together to explore an issue that can be dealt with effectively only with the combined experience and knowledge of everyone at the meeting. The search is for 57 common ground. The group meets on a "social island" for two and a half days to four days. While there are several Search Conference methodologies (Schafft & Greenwood, n.d.), the Search Conference method I considered was originally developed in the 1960s by Fred Emery and Eric Trist, and has since been extended by Merrelyn Emery (Emery & Purser, 1996). While the suggestion was made that search conferences were very possible in an online environment (O'Brien, 1998, ^|28) I could find no evidence of it having been used in that manner which has exciting possibilities for future research. Systems Thinking Using Group Model Building (GMB) Systems' thinking using Group Model Building (GMB) is a three-step process starting with structured brainstorming using the partial K J (Jiro Kawakita) technique to identify priority areas and then clustering them into 'affinity' groups. Next, the priority clusters are used by the participants to construct causal loop diagrams representing 'systems of priorities' (in contrast with list of priorities). Finally, through a group process, leverage points or key priorities are identified as the basis for a plan or decision. The systems thinking approach can be used to create consensus and the commitment to the outcome. The methodology can be applied to change management initiatives and complex decisions while the expected outcomes can be greater commitment and a shared vision (Maani, 2002). G M B was first proposed as a way to elicit simultaneous, structured knowledge from a wide ranging group of experts (Scholl, 2004; Vennix et.al., 1988). 58 Nominal Group Technique (NGT) The Nominal Group Technique has its roots in organizational planning techniques and is credited to Delbecq, Van de Ven and Gustafson of the USA, between 1968 and 1971, as the original authors (Sample, 1984). The actual date of original publication appears to be in some question, depending on the source (Jones & Hunter, 1999; Gordon 2003; Sample, 1984; Joppe, n.d. 1). What is agreed upon is that its name "...comes from the fact that participants meet as a group but do not interact in natural group fashion (a group in name only)", therefore the concept of 'nominal group' (Gordon, 2003, p. TJ1). NGT is commonly described as ".. .a consensus planning tool that helps prioritize issues" (Joppe, n.d. 1, fll). It was quickly considered to be more structured than other group-oriented techniques, such as brainstorming. Dunham (1998) suggests that NGT is "...concerned with judgmental decision making where creative solutions are sought, [and that the NGT technique takes]... advantage of pooled judgments, [where] the judgments of a variety of people with varied talents, knowledge and skills will be used together... the resulting ideas are likely to be better than those that might be obtained by other methods" (T[l and 2). Schreibman (2001) calls the NGT "...a key systems methodology used to build a structured workshop/meeting/process, including recognized consensus building capacity, usually facilitated by a 'third party.'" (fl2). The NGT is designed to facilitate collaborative and democratic decision-making (Delbecq, Van de Ven, & Gustafson (1975). According to Jones (2004), in its pure form, the NGT consists of six steps: 1. Individual generation of ideas. 2. Recording of all participants' ideas (in a round robin format). 59 3. Group discussion of all generated ideas (to organize the list and remove duplication). 4. Preliminary vote to select the most important ideas. 5. Group discussion on the vote outcomes (including additions and further merging of overlaps). 6. Final voting on the priority of items (p. 22). Jones (2004) notes that the NGT has been used in education "to investigate a wide range of topics" (p. 22). She cites studies by Boudreau (2000), (characteristics of expert associate teachers); Nelson, Jayanthi, Brittain, Epstein, & Bursuck (2002), (homework communication strategies in remedial education); and Duggan & Cox (1999), (evaluation of one-to-one teaching in general practice). NGT had also been utilized in adult education within a few years of its conception (Vedros, 1979 as noted in Sample, 1984). According to Nelson et. al. (2002), the strength of the nominal group technique is that it gives everyone a voice in planning while minimizing opportunities for a few individuals to dominate the process. This technique can accommodate large numbers of participants as well as small groups. However, in practice large groups can be subdivided into a number of smaller, simultaneous groups (Nelson, et. al., 2002). It provides professionals with clear data about participants' primary concerns in a relatively short period of time. This basic description is supported by a number of authors and practitioners (Iowa State University Extension (2001); Virginia Institute of Government (n.d.); Joppe (n.d. 1); Semple (1984); Ohio State University Extension (2000); University of Missouri Outreach and Extension (2003); Dunham (1998).) The literature indicates a number of benefits of the NGT, including: 60 • It balances participation across members. • It balances the influence of individuals. • It produces a greater number of ideas than do traditional, interacting groups. • It results in greater satisfaction for participants. • It reduces the conforming influence common to most face-to-face group meetings. • It encourages participants to confront issues on a problem-solving basis, rather than on a personal assault basis. • It leads to a greater sense of closure and accomplishment (Dunham, 1998). Also noted in the literature are challenges, as summarized by Dunham (1998): • It requires extended advance preparation, which means that it cannot be a spontaneous technique. According to Nelson et al. (2002, p. 380), "The key to conducting a successful nominal group is following a step-by-step process." • It tends to be limited to a single purpose, single-topic process. • It is difficult to change topics in the middle of the process. • It needs agreement from all participants to use the same structured method, which some people might resist. One of the warnings of the Virginia Institute of Government, (n.d.; conclusion "Uses and Abuses") was that "...during early experiences using NGT, it is most difficult for people to keep from discussing issues before all points are listed, clarified, and 61 prioritized. So, extra care must be taken by the facilitator to prevent discussion from starting too soon." It is my judgment that using NGT online assisted the process by eliminating a number of concerns, such as this one. By using email to generate the initial points and having the facilitator list these points to a discussion area online (in fact readying the list before the participant-experts enter the 'electronic room'), the impact of this issue was reduced, and discussion on points were held at bay until the appropriate time to begin. Overall, the NGT combines both quantitative and qualitative methodologies, as decisions are made through discussions as well as through voting procedures (Chappie & Murphy, 1996). Although the literature concerning the use of the NGT for my area is very limited, several studies have examined its traditional use in such areas as prioritizing goals (Booth, Struck, & Thomas, 1986), identifying problems (Hoyle, 1978), and assessing needs (Gerdes & Benson, 1995). Other than educational uses, the traditional NGT has been implemented in engineering, nursing, business, management planning, and community development (Johnson & Richards, 1983; Beruvides, 1995; Kapiriri, Robbestad, & Frithjof Norheim, 2003; Russell, 1999; and Phahlamohlaka & Friend, 2003). A diagram by Joppe (n.d. 1), which visually describes the traditional NGT process can be found in Appendix B. Given the use of the NGT for my study, two features are especially striking. First, because of time-limited participant discussion and the use of a voting procedure, the NGT is a fairly democratic process, which reduces the risk that some group members will dominate (Vella, Goldfrad, Rowan, Bion, & Black, 2000). Such a democratic process is especially beneficial when some members of a group might be perceived by other group 62 members to be of higher status, that is, more knowledgeable or more powerful in terms of academic or professional reputation. Second, the NGT makes efficient use of time: Unlike a typical committee meeting, discussion takes place only at designated times during the NGT process. The decision-making task thus could be completed in a series of online sessions. After a search of the books and journals (both electronic and print formats) and of the world wide web, I found only one instance where a classroom teacher used the Nominal Group Technique (NGT) delivered by email, as a precursor to the full classroom experience (Morrison, 1998). I discovered six research project titles in which the NGT replaced, wholly or in part, the face-to-face group meeting that usually defines the NGT, with an electronic model. Of these six references, two were unavailable, and two were deemed by me to be not applicable to my work. The remaining two were major influences on my thinking about how to implement my own research (Hoffmann, Schelle, & Yalpani, 2002; Dowling & St. Louis, 2000). In the first research study, Dowling and St. Louis (2000) examined the effectiveness of asynchronous implementation of the Nominal Group Technique as compared to a traditional face-to-face implementation. In their report, they suggest that "This paper provides evidence that computer-assisted asynchronous (CAA) implementations of the NGT are more effective than noncomputer-assisted synchronous (NCAS) implementations of the NGT - C A A implementations generate more and better ideas, and do it in less time" (p. 229). Using a quantitative, empirical, laboratory study, Dowling and St. Louis (2000) created a task (in this case, a planning task for a business environment that involved a full 63 text case study). The participants were asked, via NGT, to ".. .submit ideas to be included in this organizational policy" (p. 237). They recruited 154 first year M B A (Master of Business Administration) students and 15 M S I M (Master of Science in Information Management) students, as their subjects. In the study they assigned the students to either the C A A or the N C A S environments, with each group size being limited to between 7-10 members. "This resulted in 11 groups for the C A A environment and 10 groups for the N C A S environment" (Dowling & St. Louis, 2000, p. 237). There were two treatments for this study - the N C A S or face-to-face meeting environment versus the second environment, that of participants communicating "...over time and space using a computerized support system" (p. 237). While Dowling and St. Louis (2000) wanted to do their NGT completely online (via email), they found the following difficulty with respect to available technology and support at their institution: Originally, the intention was that the subjects would send and receive the files for this research via an e-mail package. Three issues prevented this: (1) at the time of the research, no e-mail package was available from the university which allowed easy attachment of files to e-mail messages; (2) a substantial number of the subjects did not have access to the university computer system from home; and (3) many of the subjects did not routinely use e-mail. (p. 240) To circumvent technical problems, they compromised and used a face-to-face initial briefing and end-of-treatment debriefing, while conducting the actual NGT process 64 via computer disks that were distributed to the participants in the C A A section of the experiment (p. 239-240). After recruiting the students who participated in the study (154 M B A and 15 M S I M students) the researchers divided them into 2 sections - those who took the process wholly face-to-face and those who were to do the work asynchronously. When assigned, the C A A (computer-assisted asynchronous) treated population totaled 91 students (36 female and 55 male) and resulted in 11 randomly assigned groups. (Dowling & St. Louis, 2000, p. 237) Using computer disks (which were loaded with message frames in a graphical interface) and a meeting facilitator, the C A A groups began the NGT process. The same facilitator conducted all 11 C A A meetings, met face-to-face with each group and "verbally introduced the NGT and the e-mail privacy issue. The subjects were not given a paper copy of the case, but were given a disk containing the computerized support system, the text of the case, and a schedule Of deadline dates for the different stages of the NGT. . . [they] were asked not to discuss the case with anyone else..." (Dowling & St. Louis, 2000, p. 239). Table 4 presents the plan for the C A A version of NGT. 65 Table 4 CAA Stages in the NGT Study N G T Stage Group 2: C A A 1. Idea generation 2a. Discussion 2b. Discussion (continued) 3. Voting 4. Final Individual idea generation Lists given to facilitator who collates ideas into one list. Each member looks at other members' ideas, suggests combinations, asks for clarification, and adds new ideas. Facilitator collates lists and makes any changes recommended by majority. Each member is invited to respond to questions from the previous stage. Facilitator makes any changes recommended by the majority. Each member votes for five ideas from the revised list of ideas. Facilitator tabulates votes and records results on disks. View results Post experiment questionnaire Debrief with facilitator (Dowling & St. Louis, 2000, p. 238) In their study Dowling and St. Louis (2000) hypothesized a number of statements and ran the data through a various statistical tests, including M A N O V A , A N O V A and multivariate Mests. They concluding that there is ".. .strong evidence that, for planning type tasks, C A A implementations of the NGT are more effective than N C A S implementations of the NGT. They generate more and better ideas, and do it in less time" (p. 244). They go on to recommend further research in the area: "Additional task types (beyond the planning task) with appropriate processes...need to be studied" (p. 244). 66 Dowling and St. Louis's work was important to my study for several reasons. It was the first study that tried to use NGT in an electronic way. It formed the basis for the more significant study to inform my work, the G P M Deutsche Gesellschaft for Project Management, discussed below. They used small groups to conduct the larger NGT process, which was useful to my own methodology. They offered recommendations that were useful for me and came to conclusions that assisted me in my decision to use the NGT online. The second study significant to my research was conducted by the G P M Deutsche Gesellschaft for Project Management, which "...conducted an international survey of experts on the future of project management by inviting 25 experts of project management [from] around the world to take part in a survey using an asynchronic, internet-based realization of the 'Nominal-Group-Technique' (NGT)." This "internet survey project" took place from February to May, 2002 with the primary goal and basic theme of the expert survey to discuss the following question: "What developments in the field of project management can be expected in the next ten years, in your opinion?" (Hoffmann, Schelle & Yalpani, 2002). In all, 23 international participants were listed as having taken part in the study. Profiles of participants were included, indicating that they represented nine countries. Participants were emailed in advance of the beginning of each phase of the study, and this helped to keep them involved and committed throughout the nine weeks of the online process. Participants completed all of their interactions on the Internet, without any face-to-face meetings. A l l interactions were done on their own time schedule, within the given period allotted for each phase. 67 In this project, the authors used an initial letter to introduce participants to the research, the process, the researchers, and their expectations regarding participation. They also included a brief explanation of the software/website to be used, and they encouraged participants to "become familiar with the program" (Hoffmann, 2003, p. 3). They chose to conduct the NGT in four phases, reducing the typical NGT process by two phases. They indicated that each phase would be conducted over a three-week period, with the exception of Phase 4, which had no end date to allow the final debriefing and discussion greater flexibility. Within each phase, they indicated to participants what the expected personal commitment would entail. Of course, the time each participant will need... can only be estimated. For Phase 1 (Proposing the theses) you should calculate about 1 -2 hours (over a 3 week period)... naturally more time will be needed if you present more than 1 thesis. Phase 2 (Reading and commenting on the theses) should also take about 1 -2 hours (over a 3 week period), for Phase 3 (Evaluation) you should also estimate about 2 hours (over a 3 week period), and for Phase 4 (Results) you will need approximately 1-2 hours (or more, i f you like to study the results more thoroughly). (Hoffman, 2003, p. 1-2) In personal email correspondence with Dr. Karsten Hoffmann (full text of the correspondence can be found in Appendix D), he indicated the following (where possible I left full text as emailed with only some adjustment for the English to clarify an occasional point): 68 We have learned in this first version of the survey, that: • We need better tool support to see how often and at what time each of the participants had made his/her last login on the project web site. • We should use only four phases, to make the survey more efficient. • We should start each phase with a new mailing to all of the participants (altogether we needed more active guidance for our participants). • We should remember [to contact] some participants to fulfill their work, because the whole project was often waiting for one or two responses. • The [re] was no trouble with the principle of the Nominal Group Technique, because all theses were anonymous until the last phase of the survey; but it needed a lot of time to bring all work from all participants together (Janes, 2003b). This Project Management study was.a key to my own work with the online NGT. It was the only example I had to follow as a possible model and one of the authors was most generous in his sharing of his experiences. The lessons learned by Hoffmann and his colleagues were valuable in pointing out potential problems and possible solutions to issues that arose during my own NGT implementation. The Opportunity In their current work, e-Research, Anderson and Kanuka (2003) note: ".. .there have been a few exploratory studies testing the efficacy of Net-based Nominal Group Technique" (p. 126). They note Pazos, Perazzoli, Jiang, Canto, and Beruvides (2001) 69 work and suggest that the researchers "...found no significant difference between the amount of ideas generated nor the satisfaction with the process between Nominal Groups conducted face-to-face, and those that used a text chat environment on the Net" (p. 126). This statement is supported by the work of Dowling and St. Louis (2000) who note that "There is no statistically significant difference in satisfaction with respect to the opportunity to contribute, the time required to reach a decision or the decision quality" (p. 242). However, as noted earlier, these authors do indicate that the electronic NGT used by them was more effective than the face-to-face NGT in that more and better ideas were generated, and in less time. Anderson and Kanuka (2003) also suggest: A finding of no significant difference is interpreted positively, as the Net-based approach offers convenience, accessibility, and cost savings compared to face-to-face groups. We anticipate in the near future that the availability of voice input and discussion will result in increased use of this form of structured real-time consensus making for both research and less formal decision making, (p. 126) Given the lack of formal research in using Nominal Group Technique online (outside those few research reports noted), and the assertion of Anderson and Kanuka (2003) as to its efficacy, the NGT appeared to be a methodology that suited my research intentions. It was also ready for further exploration, as a model for conducting consensus research wholly online. 70 Conclusion of the Methodology Decisions There are a number of reasons for the selection of the NGT as a methodology for my study of constructivism and how it might be manifested in online instruction. First the NGT is "a structured meeting which seeks to provide an orderly procedure for obtaining qualitative information from target groups who are most closely associated with a problem area" (Vande Ven & Delbecq, 1972, p. 338). Second, I required an approach that would be, at the same time, not too time consuming but which would be comprehensive in terms of its results. NGT met these criteria. I was also interested in the characteristics of the NGT that would lend themselves to the online environment - that it was a group activity, that it could be conducted in a reasonable timeframe, that it promoted a non-critical atmosphere during the discussion stage, that it was a structured format, that it promoted better quality of ideas, and that peer influence was likely to be reduced, if not eliminated (Gibson & Soanes, 2000; McMurray, 1994). Finally, I was interested in a consensus methodology that supported the active involvement of a facilitator. I was interested in being a part of my research, as opposed to distant from it, throughout the whole process. Considering that I was going to conduct the NGT online, and online asynchronous discussions are best supported by a facilitator (Dowling & St. Louis, 2000, p. 232), the NGT was a good choice for the research work I was about to undertake. 71 Plan of Action for the Online NGT Building on the research of Hoffmann, Schelle, and Yalpani (2002) and Dowling and St. Louis (2000), I used a fully electronic implementation of the Nominal Group Technique. Through the use of an online NGT, this study identifies the principal tenets/presuppositions of constructivism as a theory of learning, and suggests processes or instructional approaches that flow from this theory for the development of online learning. The research was conducted over approximately five months, from March through September 2004. Site and Setting The 'setting' for this research was at first email (facilitator/researcher to participant and participant to facilitator/researcher), and then a move to an online discussion environment, where the discussions took place. Participants also had access to a voting area, again, anonymously, where they were able to conduct their ranking of the ideas and issues that resulted from the brainstorming in Round 1. This environment was open to the participants by password only, and remained accessible online for the length of the research. The online environment also housed the preliminary documents, copies of the materials used during the process and any other materials deemed necessary during the research period. Initially, the WebCT environment was considered as the shell for the online discussions and voting process. WebCT is a course delivery platform that has the ability to have online asynchronous discussion forums created within the course tool. It could also handle the voting and survey processes intended for this research. Finally, many of 72 the participants I approached were very familiar with this platform, because of its wide use in educational environments. Using an already existing platform made more sense than creating a new environment. Eventually, WebCT was used for Rounds 1, 2, 2.2, and 4. A web-based commercial survey tool (HostedSurvey) was selected for Round 3 and for the final evaluation of the process. The selection of the second tool was made on the basis of robustness: the quiz/survey feature of WebCT lacked the sophistication of HostedSurvey. Group Size Throughout the literature the group size of the NGT is a debatable issue, with traditional nominal groups displaying a broad range, from small (10 participants) to large (170 or more participants). However, those using large groups frequently formed a number of smaller subgroups for the NGT process (Fuller, Gaskin, & Scott, 2003). Both examples of computer-based NGT had small participant pools ranging from 10-23 (Dowling & St. Louis, 2000; Hoffmann, Schelle, & Yalpani, 2002). Keeping the participant size to a small group is confirmed by the findings of Lomax and McLeman (1984), who indicated that widening the group changed the process to a degree which might bias and nullify the results. Anderson and Kanuka (2003) confirm that "Generally, smaller size groups are involved in Nominal Groups than in other consensus techniques" (p. 126). It was my intention to offer the opportunity to participate to more than 50 participant-experts, in the expectation that I would be able to form a nominal group of approximately 30-40 committed respondents. I worked on the belief that up to half of 73 those approached might not commit to the final process (Anderson & Kanuka, 2003, p. 129). The Participant-Experts The participants were experts, and were chosen in accordance with the following criteria: • They were either current or retired adult educators, educational technologists or instructional designers, or current masters or doctoral students, and had one or more of the following characteristics: • A research history of activity in one or all of the above areas. • A demonstrated knowledge of constructivism in their work or practice via published writing or conference proceedings. • Had taught online or developed online learning experiences sometime in the past 5 years. • Had researched or written about the online environment and/or learning theories sometime in the past 5 years. A l l had to be willing to participate fully in this research for the length of the research timeframe. The participant-experts were identified using a number of sources. Beginning with colleagues, mentors, and identified leaders in the field, I created a list of potential participants. Once this list was exhausted, I searched the current literature and posted requests for participation on current listservs and bulletin boards likely accessed by people in related disciplines. 74 The Facilitator A majority of NGT studies reported in the literature used one facilitator to conduct the NGTs, even those with multiple groups, for two reasons: (a) to ensure a standardization of the procedure across all groups, and (b) for "...simplicity in administrative procedures" (Fuller, Gaskin, & Scott, 2003; Lomax & McLeman, 1984). For the purposes of this research, I as researcher acted as sole facilitator throughout the NGT process. Being sole facilitator allowed me to keep detailed records of all participation across the multiple rounds of the study, and permitted me to act swiftly in the event that any problems arose. It also allowed me to become intimately involved in and knowledgeable about, the NGT process itself in an online environment - something that is of benefit in terms of building knowledge of the potential use of this technique in an online environment, in the future. Schedule of Events The following summary reflects the step-by-step process which I used in the design, implementation and follow-up of the online NGT. Preliminary Documents and Process (Fall 2003) • Set criteria for participant-experts. • Identification of participant-experts. • Initial contact of potential participants re their willingness to participate. • Investigate potential software/platforms for use in the NGT. • Select technology and design web site. 75 Getting the NGT Started (Winter 2004) • Email participant-experts documentation on the upcoming NGT, background information on the NGT process, explanation of the structure of the NGT, the participant/facilitator expectations, and anticipated commitment of time. • Set up the environment where the data collection/interaction will take place. • Contact participant-experts by email. • Confirm participant-experts by sending/receiving signed Letters of Consent to participate in research, (see Appendix A for Letter of Consent) • Pre-test the NGT research questions with colleagues. • Test the online environment with colleagues willing to participate. • Record journal of the decision-making and process development. • Create anonymous login IDs/passwords for participant-experts. • Confirm emails used by participant-experts and send them an email to begin. Implementation Process (Spring/Summer 2004) • Begin email brainstorm (Round la) with the first question, following with second and third questions. • Synthesize the returned data, all three questions - create lists (Round lb). • Post lists gained from brainstorming (Round 2a). • Begin online clarification, modification and additions/deletions of ideas (anonymously conducted) (Round 2a). • Synthesize returned data via discussions (Round 2b). • Post synthesis of discussions (Round 2b). 76 • Conduct an additional round, if necessary. • Begin voting process (Round 3a). • Synthesize voting results (Round 3b). • Online participant-experts agree with the vote and participate in final discussion (Round 4a). • A final vote is taken (Round 4a). • Close the online discussion (Round 4b). • Thank participant-experts online, as group (Round 4b). • Synthesize final data and collate the process/results (Round 4b). • Keep journal of the process and thinking. • End of process survey to NGT participant-experts to ask them about their perception of the process. NGT Questions for the Participant-Experts The following questions to the participant-experts were taken from the primary research questions and were modified for the use of the participant-experts within the NGT experience. The NGT questions for the participant-experts were: • What do you believe are the primary features, characteristics or elements of constructivism as a theory of learning? • What approaches or procedures to instruction in relation to online learning do you believe flow from constructivism as it is currently perceived? • Given these characteristics and procedures, what, do you believe, is the role of the instructor in a constructivist learning environment? 77 Definition and Measurement of Agreement In a consensus methodology it is important to define what 'agreement' is when describing consensus. According to Jones and Hunter (1999), agreement can take two forms in the consensus process. The first form is the "extent to which each respondent agrees with the issue or statement under consideration..., and second, the extent to which respondents agree with each other - the consensus element of these studies..." (Jones & Hunter, 1999). "Most dispute resolution professionals believe that groups or assemblies should seek unanimity, but settle for overwhelming agreement that goes as far as possible toward meeting the interests of all stakeholders" (MIT-Harvard Public Disputes Program, n.d., p. 4). Jones and Hunter (1999) also remark that it should be made clear to each participant that they need not agree with the group view, although in the NGT it is often the case that those holding alternative views to the rest of the group may face exact questioning of their views from the other participant-experts. For nominal groups, rules have been developed to assess agreement when statements have been ranked on a 9-point scale. For example, participant-experts ranked appropriateness of intervention, when 0 = inappropriate and 9 = appropriate. The scale was then broken down so that scores were assigned as 1 -3 = non intervention, 4-6 = equivocal, and 7-9 = intervention (Jones & Hunter, 1999). Jones and Hunter (1999) go on to suggest that there are two rules when assessing agreement. The first is strict agreement, where all ratings fall within one of these predefined regions. The second is 'relaxed agreement', where all ratings fall within an acceptable range but the "...group 78 opinion is ambiguous as to whether intervention is indicated or not" (Jones & Hunter, 1999). In my research study, for Round 3, the vote, I implemented a three-sided matrix with three distinct point scales. For each idea statement, I asked participants to • Agree/Disagree that the idea statement 'was a primary feature of constructivism and online learning; • Rank the 'importance of the statement' to constructivism and online learning (1 = essential; 2 = very important; 3 = important; 4 = somewhat important; 5 = not important); and finally to • Indicate i f they felt the idea statement was a personal view (held personally - a minority view) or a global view (a commonly held belief about constructivism and online learning - a majority view). Data Analysis My methodology had both a qualitative and a quantitative dimension, and both needed to be accommodated in the analysis of the data from the NGT process. The process weaved from qualitative to quantitative, and back to qualitative, as the various rounds were completed. Not only was consensus via voting a goal, but also the individual comments and belief systems expressed by the participant-experts were an important element in the final data analysis. The deductions of the participant-experts were reviewed in light of the research questions, and conclusions drawn regarding the consensus on the issue of constructivism and its implications for online course design and 79 the teaching of online courses. In addition, the utility of the NGT, as on online research methodology, was evaluated by the facilitator. Ethical and Safety Issues I have conducted research at the master's level and have taught and facilitated both face-to-face and online since the late 1980s. With this comes an empathy and sensitivity on my part to participant-experts' needs and feelings while they engage in online activities. Participant-experts were selected and their online/email participation observed and synthesized only after they gave full, informed, and voluntary consent. Individual and group data collected during the data collection process was not discussed within the educational technology community, to avoid any personal or professional risk to the participant-experts. The transcripts of the participants' email/online involvement were stored separately electronically, and were given numbered identifications to further ensure confidentiality. I kept a list of corresponding numbers and names in a separate location. Only pseudonyms were used during all online discussions. Data were stored according to The University of British Columbia guidelines and will remain, in CD format, at U B C for a period of 6 years before being destroyed. The names of participant-experts and their institutions, and other personal or confidential material, were not used in reporting the research results, nor will they be disclosed in the future in professional correspondence of any sort, unless permission is granted. Participant-experts were made aware they were free to withdraw from the study at any time, with no obligation or adverse consequences. If any participants choose to 80 withdraw, their data would be immediately deleted and any information obtained from them would not be used. Participant-experts were made aware that they could make a formal complaint to The Office of Research Services at UBC, through the contact numbers provided on the Letter of Consent (see Appendix A) and through a link on the NGT web site, should they disapprove of the conduct of the study. Furthermore, there were no apparent power issues that were thought to compromise the study, or to lead to any harm for the participant-experts. 81 CHAPTER 4 THE NGT PROCESS A N D ITS EFFICACY ONLINE This chapter describes the NGT process as implemented in my study, including the choice of delivery platform, the selection process for participant experts, their commitment, and the activities associated with getting the NGT underway. It also describes the NGT process as it progressed, from the first day interaction with the participant-experts to the synthesis of their brainstorming, the voting processes and the final discussions and consensus. It also presents participant opinions on the process, my conclusions about the efficacy of the NGT in an online environment and as a research methodology, and specific recommendations for future implementation. Platform Choices In implementing the NGT online, I made the decision about platform with little direction, since there was very limited discussion in the literature on the use of the methodology in that environment. I was faced with making a choice based on information from just a few researchers regarding their choice of platform, plus on my own and colleagues' knowledge of various options. Hoffmann, Schelle, and Yalpani (2002) chose to use as their online environment a "web platform [using]... php3 (script language) and mySQL (database)" based on the work of a colleague (Janes, 2003b). This created a number of issues which they identified as needing to be improved including ". . . better tool support to see how often and at 82 what time each of the participants had made his/her last login on the project website" (Janes, 2003b). As I examined the possibilities - html pages, choosing a platform that I was familiar with such as WebCT for the discussions, or creating a collection tool from scratch, as had been done by Hoffmann et al. (2000) - 1 concluded that I wanted to keep the interaction and data collection as clean and smooth as possible. Overall, I needed a site that fulfilled the following requirements: • Allowed for instructions to be presented during each round. • Allowed for password protection during each round. • Allowed for interaction with an online discussion area that could be both private (available only to a single participant as in Round 1), and public (available to all participants as in Rounds 2 and 4). • Allowed for the movement to a survey tool (as in Round 3). At approximately the same time as I was making platform decisions, I participated in an online Delphi study being conducted by faculty members at the University of Prince Edward Island (UPEI). This study, while not an NGT methodology, was similar to what I had envisioned. Basically using html pages as entry points, the researchers used WebCT discussion forums for the discussions, and a software program called TestPilot for the survey work within the Delphi. After email conversations with the lead researcher, I determined that they were happy with performance of the WebCT component, but they were not as satisfied with the survey tool they had chosen. They found "...Test Pilot is a bit of a [problem] when it comes to data analysis; e.g. items are 83 not numbered the same as they are in the survey, and every once in a while, Test Pilot truncates a response (nobody has figured out why yet...)" (Janes, 2003c). Given their difficulty with Test Pilot, I sought an alternative for the survey components of my study, and contacted a colleague at the University of British Columbia with whom I had worked with over Fall, 2002 and Spring, 2003 on another survey research project (Janes, 2004). That project had used the survey software Hosted Survey (http://www.hostedsurvey.com/), and my colleague indicated to me that it performed all of the same tasks as Test Pilot, and did so without the additional problems noted by the UPEI researcher. Upon investigation, which included email correspondence with a representative of the Head Office Customer Support Unit, I discovered that Hosted Survey offered me the kind of support and data analysis options that would suit the NGT online (Janes, 2004a). With my criteria in mind, as well as the experiences of several others I had consulted (Janes, 2003c, 2004, 2004b), I clarified what I needed optimally, and which tools could enhance my data collection procedures. Table 5 provides summary information on each platform/tool. 84 Table 5 Comparison of Platform/Tool Features Considered for the NGT Feature Hosted Survey Test Pilot WebCT H T M L page Cost No cost* Standard pricing ($750 US) No cost** No cost** Time Quick and easy Quick and easy Technical support used Technical support used Password Protection Yes Yes Yes Yes Track Participants Yes Yes Yes Unknown Compile Numerical Data Yes Yes Yes Unknown Allow for Qualitative Data Yes Yes, with problems Yes, if quiz tool used Unknown *Hosted Survey does have a price scale, but it offers up to 250 responses free - which in the case of my study resulted in no cost. ** Both WebCT and H T M L hosting were supplied by the Distance Education and Technology Unit of the University of British Columbia, as support for graduate research and as a contribution to my thesis. Having chosen to work with the WebCT discussion tool, and to use Hosted Survey as a tool in the case of my survey functions, my next step was to contact a technical support advisor who would create the virtual online environment that would house my NGT. I contracted with Alan Doree of Distance Education and Technology at the University of British Columbia (Janes, 2004b), who proceeded to create my online environment between January and March 2004. By March 15, 2004 the site was online and ready for use. (see Figure 1.) 85 J Welcome to this online study ConstructMsm and instructional Design: An Exploration Using an Asynchronous Online Nominal Group Technique. The purpose of this study is to identify what you as participant-experts believe are the tenets/presuppositions of constructivism as a theory of learning, and to suggesl instructional approaches that flow from this theory for the development of online learning. Using a consensus-building methodology, the Nominal Group Technique (NGT), wholly online, the goal is to reach a consensus among the participants on what constructrvism means and its implications for the design of online learning programs. You have been asked to participate in this research because of your expertise and experience in the area of learning theory, educational lechnology and/or online learning. Please review the information under the "Confidentiality', "Contacts' and "Research Team' Links before beginning. They can be found at the lop of each page. The Home' link brings you back to the homepage To begin the study, click on the link below. The Nominal Grmip Tt-chmryie Instructional D e s i g n Figure 1. Homepage to Online Discussion Website for Study Setting up the Online NGT The NGT web site was password protected, which required a decision regarding the use of either a gateway for login (with the same look/feel as the site), or a dialogue box to ask for login and password. In the latter case, once participant-experts accessed the site, the dialogue box appeared after they loaded the first page, either before they clicked on the Nominal Group Technique link or once they click on the link. After consulting with my technical support person, we agreed to a login and password space as the participant-experts entered the private or group discussion forums. Each participant was assigned to the login and password, and remained anonymous to each of the other jg£|fc*£f»fi^|^ 3ywaosaft word-rhe... f ^Constructiveanande... iS; Osratnaltasm - Net $... 86 participants. Only the facilitator of the NGT was able to match the login with the identity of the participants. To maintain some control over the progression of the Rounds, I decided to block each of the Rounds not currently in use. Hence, at the beginning of the study only the Round 1 link was visible to the participants. The initial intention was that each Round would remain open for the one week that it was active, and then be closed. This pattern was to be used for each of the four Rounds. Since Round 1 began on a Thursday, I had anticipated that each Round would close/open on each successive Thursday until Round 4 was completed. The final activity, the N T G satisfaction survey, was to open at the end of Round 4. When open, each Round was to be accessible to the participants by the same login and password they had been issued in the first Round. It was my intention to provide login and password repetitions, as reminders to participants, in all follow up and Round notification emails to the participants. However, as the Rounds progressed, it became obvious that the time line for Rounds needed to be extended, because of the busy schedules of participants and the . increase in the participant task-load because of the sheer amount of data. Also the time line for the NGT process needed to be extended because of the requirement for greater time allotment between Rounds to analyze and organize the data for the next Round, on my part as facilitator. 87 Data Forum In addition, to the private and public discussion forums, I created a forum, which was only visible to me as facilitator/researcher, labeled Data. This forum was originally set up to allow me to begin to synthesize the discussion forum data as the participants were creating it. It also allowed me to access the data from any location where I might be during the study. Thesis Site Backup In discussion of this issue with my technical support person, I asked that he ensure the discussion forums and data for each Round was backed up (either on his computer or on CD). I recommended this task be done several times over the week of each Round, and suggested that it minimally be done at the closing of each Round, and midway through each Round. I also made use of the compile tool within WebCT to keep copies of all of the discussions, both private and public. Participant-Expert Selection Concurrent with the decisions on how the online NGT environment would look and function, was the issue of identifying the possible participants for the online NGT. The participants were to be experts and chosen based on the criteria described in Chapter 3. I began my quest by seeking participants among my own colleagues and professional contacts that I had developed over more than a decade of professional experience in distance/online education and instructional design. I invited these contacts 88 to participate and also to recommend others who met my criteria and who might be willing to participate. I had an initial list of 120 possible participant-experts. From that initial list, I culled those who did not meet my criteria, or whose contact information was inadequate. I eventually developed a working list of 78 names. On March 31, 2004 I emailed each of the 78 individuals on the working list, inviting them to join my study and to return the embedded consent form (see Appendix A), to me within a week, signed, as an indication of their willingness to participate in the study. The signature could take the form of a faxed document to my office, or it could be returned to me by email with their name and date of agreement typed into the returned document. Over the next few days I received 16 consent forms. Two respondents also wrote to me to suggest an additional three colleagues who would be eligible and potentially interested candidates for participation. Upon investigation, I agreed with their suggestions, and followed up by emailing three additional invitations. Thus my working list at this stage of the participation selection process was 81 potential invitees, of which eight proved to not have working email addresses, and on follow-up proved to be impossible to contact, for a total of 73 eventual invitees. Table 6 presents summary information on the invitations issued and the responses over a two-week period. 89 Table 6 Participant Selection Process Date Activity Number March 31 Invitations sent 78 April 9 Positive responses 28 April 9 Non-responses 12 April 9 Bounced invitations 9 April 9 Repeat Invitation to non-responses 28 April 14 Additional positive responses 8 April 19 Additional invitation and two positive 2 responses April 22 Total participant-experts 38 In actuality, not all of the 38 participant-experts took part in Round. 1. Four participants lagged behind in posting to that Round, and two participants were added late in the Round. However, these six participant-experts were contacted and informed that they could move on to Round 2. It was understood that the emergent design of my study permitted some flexibility in terms of participation^ and "full participation in the brainstorming phase is not necessary for the integrity of the method" (Janes, 2004c). Participant Demographics When the study began on April 15, 2004 there were 36 participants. Within the next week two additional participants were added, for a total of 38. Given my criteria for participation, the demographics were as anticipated. Demographic information on participant-experts is presented in Table 7. 90 Table 7 Demographics of Participant-Experts Participant-Expert Feature Number Number of countries represented 9 Number of continents represented 4 Level of education - completed doctorate 25 Level of education - graduate degree 13 Gender - female 19 Gender - male 19 Employment - university academic appointment 17 Employment - university professional appointment 15 Employment - polytechnical appointment 1 Employment - private educational consultants 2 Employment - K-12 distance education 1 Employment - research professional 1 Employment - educational software developer 1 Implementation of Round 1 Once preliminary preparations for Round 1 were completed, that Round officially began on April 15, 2004. It began with a welcome message posted by me in the discussion forum. A WebCT tutorial was available for those not familiar with the software or discussion tool, and was linked to the NGT Homepage, under How to Proceed. Conscious of the heavy workload of many of the participant-experts, I had decided to send a reminder to post ideas on Sunday (mid-way through the Round), and again on the final day of the Round. By the mid-way point of Round 1 just six participants had posted, having spent on average fifty-three minutes addressing the questions set for that Round. 91 By using the WebCT discussion forum, I was able to track activity within the site (as suggested by Hoffmann, 2003), and I noted that an additional four participants had logged into the study web site, but had not contributed to the postings. The inactivity, on the part of the remaining participants, prompted me to issue a gentle reminder email to those participants who had not as yet posted or contacted me. Within two days of my reminder, an additional twelve participants had posted, with approximately half of the participants still absent from Round 1. Following a second gentle reminder, eight participants posted, and another three visited the site. Round 1 participation closed on April 22, eight days after it had begun, with a total of 32 participants having participated, for a response rate of 89%. Analysis Activity Between Rounds 1 and 2 Preparation of documents and procedures for Round 2 involved compiling the ideas in the Round 1 postings. I needed to devise a structure that would allow participants to comment on what had been posted (anonymously) by their fellow participants, but that did not require judgment on their part. I also needed participant-experts to identify any key ideas that might have been missing from the summary list. In dealing with the ideas posted, I used the idea of emergent themes (Morrison, 2004, p. 66), which consisted of creating clusters of sub-themes to organize the data. Grouping the themes and sub-themes into common segments would allow the participants to begin the process of decision-making on agreement/disagreement with the ideas expressed, and with their wording. It would permit the addition, blending or deletion of themes and sub-themes during Round 2. 92 Another issue arose as I created the list from the Round 1 brainstorming, activity: the issue of group size. Round 1 had 38 participants (32 of whom were active), and a full and open discussion becomes very unwieldy with such a large group. The ideal group size for unstructured and open discussion is approximately 20 maximum, and if necessary it is advisable to split groups to avoid exceeding the maximum (Palloff & Pratt, 1999, p. 155; Nelson, et. al., 2002). Following discussion with my supervisor (see Appendix F), I made the decision to create two separate groups for Round 2 (named Group 2A and Group 2B). It was initially thought that these two groups would stay together for all of the remaining Rounds. Each group - 2A and 2B - would work on a list of items that had been generated only by members of the particular group members. I created three MS Word documents, one for each Round 1 question, which listed the items noted by the individual contributors. I merged items deemed to be similar, reducing the original 400+ idea statements to 360 idea statements. Following on the work of Hoffman (2003), Round 2 allowed questions about any of the individual ideas to be posed, for clarification purposes. The goal here was to reach a common understanding of the content of the idea statements within each group. The participant posing the particular idea statement (who remained unknown to the group) could comment on his/her own submission, but was asked not to do so, unless it was to answer a question of clarification. The participants were told their "task [was] to seek clarification, through a group discussion, on any of the items on the list that you find to be vague or unclear. This is also an opportunity for you to suggest points that should be merged, deleted or added to the list, i.e. new points" (See Appendix E for full email text). Group members could, during this round: 93 • post a question for clarification to any participant; • post a recommended merger of points (using the numbers assigned to a points as reference); • suggest the deletion of a point (again referring to the points assigned number); • add a point that they felt was missing from the list. Implementation of Round 2 . . An email was sent on April 26, 2004, to each participant (see Appendix E) explaining Round 2 and its intentions. I also extended the Round 2 timeline by two full days to provide more time for participant-experts to work through the tasks for this Round. In Round 2, the synthesis was much slower than anticipated. Round 2 was originally to run until May 7th, however a major problem surfaced. There were too many ideas to synthesize, (as one participant noted - overwhelming). One participant recorded over three hours reviewing the idea statements, in messaging another about the study. Finally, as the Round was reaching closure, I decided to extend this phase into Round 2.2, with the approval of my supervisor (see Appendix F). The intention was to reduce the number of idea statements to a workable set, hopefully less than 20 per group, in just one category for each set. Because some of the eight original categories were quite large, containing 50-80 idea statements, it was decided to break the larger categories into more than one idea set. In all, 11 small groups were formed, and participant-experts were randomly assigned to each group. The final breakdown of groups and categories/idea sets, for Round 2.2, is presented in Table 8. 94 Table 8 Participant-expert Groups and Category/Idea Statement Breakdown Group Name/Category Number of Idea Statements Group 22 A : Learning Environment (A) 21 Group 22 B: Activities and Constructivism 34 Group 22 C: The Learner and Constructivism 34 Group 22 D: Learning Environment (B) 22 Group 22 E: Instructional Design 29 Group 22 F: Instructor and Constructivism (A) 25 Group 22 G: Instructor and Constructivism (B) 25 Group 22 H : Instructor and Constructivism (C) 28 Group 22 I: Social/Community/Culture 13 Group 22 J: Assessment and Constructivism 8 Group 22 K : Theory and Constructivism 18 Round 2.2 opened on May 14, and it was scheduled to run for 11 days. This Round was originally to run from May 14-24, with a view to Round 3 beginning on or about May 29. By May 23, eight of the eleven small groups had been accessed, and five of these eight groups had been accessed by only one of the four participant-experts assigned to the groups. This was a cause for considerable concern. On the evening of May 23,1 sent out three versions of the Round 2.2 gentle reminder. Each version extended the round to May 27, but was directed to the participant-expert depending on past involvement (having not logged in at all; participated only in Rounds 1 and/or Round 2; already posted in Round 2.2). This strategy appeared to work. Within one day, the Round 2.2 activity had increased, with only one group remaining inactive. A number of participant-experts emailed me as facilitator to indicate their commitment to post before the Round ended. They noted a number of issues that intervened in their desire to post, including family 95 commitments, work deadlines, conference travel, other commitments, holidays and the like. Additional requests to extend the Round were made, and as a result an additional ten days were added to Round 2.2, which eventually closed on June 7. At the close, all 11 groups had submitted a final list for consideration in Round 3. Analysis Activity Between Rounds 2 and 3 I spent approximately two weeks synthesizing the data from Round 2.2. Items deemed similar were combined and/or deleted, and language was adjusted to make the items into idea statements where necessary. At this stage the original item statements and the final item statements were reviewed by a fellow researcher, to ensure that all pertinent idea statements were captured. As well, ongoing work on creation of the online web survey within the HostedSurvey toolset was undertaken. The final e-survey instrument resulted in 51 idea statements noted under 10 categories, as follows (see Appendix G for the survey instrument): • Constructivism and knowledge • Constructivism and active participation • Constructivism and authentic learning • Constructivism and multiple perspectives • Constructivism and self-directed learning • Constructivism and collaboration • Constructivism and community 96 • Constructivism and the instructor • Constructivism and instructional design • Constructivism and assessment Implementation of Round 3 On June 22, 2004 an email invitation from HostedSurvey was sent to the 38 participants in the NGT, informing them of the beginning of Round 3. Round 3 required that participants vote on the 51 idea statements that emerged from Round 2.2. They were asked to indicate the importance of the each idea statement on two scales: a simple two-point scale of agreement or disagreement on whether the idea statement represented a core meaning of constructivism, and a five-point scale indicating the idea statement's placement - from essential as a descriptor of constructivism to non-essential. Also, on a third scale, they were asked to indicate whether they believed that each statement represented a global view of constructivism, or their personal view. At the end of the first week of Round 3, 13 participants had begun the survey (and 11 of them were complete). Of the two incomplete surveys, one participant contacted me about server problems during the completion, and completed the survey early the following week. The other participant did not contact me: I contacted the individual directly but did not receive any response, so that survey remained incomplete. I sent a first gentle reminder message to participants on Friday, June 25, and a second gentle reminder message on June 30. In addition, as with all previous Rounds, individual participants emailed me to offer encouragement, to indicate their willingness to participate once their time freed up (before the end of the Round), and to note that they 97 would be away (i.e. email away messages automatically generated). In all, when Round 3 ended on July 10, 33 respondents had completed the survey, for an overall response rate of 87%. Analysis Activity Between Rounds 3 and 4 Results of the vote in Round 3 were tabulated, and data were displayed in the form of frequencies and percentages. In addition to the voting data, three threads were developed by me to guide the final discussion in Round 4. • Thread 1: Comments on the Vote. Any final comments on the vote for the idea statements were encouraged • Thread 2: Consensus and its Meaning. Participants were to provide their opinions on what constituted consensus, in terms of percentages. • Thread 3: Affirmation of the Vote. Participants were to confirm the votes and the final definition of consensus. It was anticipated that these focus threads would make the final discussion meaningful to and manageable for the participants. Implementation of Round 4 Round 4 began on July 18, 2004 with invitations to participate sent to all 38 participants via email. This Round, using WebCT for the discussion area, was the site for the final conversation among the participants. A l l participants were invited to a general 98 public space, where they could view the postings of each other, although of course the pseudonym for each participant was used. In all 21 participated in the final round for a response rate of 55%. The lower participation rate at the end is possible because of the length of the process and its overlap with the traditional summer vacation period for most academics. A gentle reminder message to all participants was sent out on July 27, and again on August 19. The second gentle reminder also noted to participants that the final task, the end-of-the process survey was about to be launched. End of Process Survey The End of Process (EP) survey was implemented to obtain the views of participants regarding the online NGT process itself, based on their experiences as participants. The survey was evaluative in nature, asking for responses in a number of categories, including demographic information, time committed to the study by participants, their overall impression of the process as a research experience, the utility of WebCT and HostedSurvey, and their impressions on the various Rounds and how they worked. On August 20, final testing for the End of Process (EP) survey was conducted. HostedSurvey sent the first invitation to all participants on August 20, 2004. Two gentle reminder messages were subsequently sent: the first on August 25 to the 21 participants who had not yet completed the survey, and the second on September 1, to the remaining 14 participants who had not yet completed it. A separate message was sent by Hosted Survey to two participants who had begun the survey but not finished it. In all, the End 99 of Process survey time line was ended on September 5, 2004 with 27 participants having completed the survey, for a 71% response rate. 100 CHAPTER 5 D A T A A N A L Y S I S This chapter presents a summary of the data, which was generated over four rounds of the NGT, and through the Process Survey which followed the NGT. Data are, for the most part, qualitative, but voting data are presented in quantitative form. The NGT process, in the early discussion rounds, resulted in massive amounts of data hence only key data are presented and discussed here. The complete data set is found in Appendix I. Round 1 Data The goal of the first round was the identification by the participant-experts of the ideas or characteristics that they believed applied to the three questions posed around the basic theme of the research, constructivism and online learning: • Primary features, characteristics or elements of constructivism as a theory of learning; • Constructivism as potentially applied to online learning; • Role of the instructor in implementing an online constructivist learning environment. The task, for participants, involved posting their initial ideas and defining elements of constructivism. In all, over 400 initial idea statements were generated. These idea statements were embedded in prose postings. My role as facilitator, in dealing with this large data set, involved initial extraction of all idea statements from the prose 101 responses. This involved reading each prose response for semantic content, identifying the idea statements and comparing them with other idea statements, deleting repeated idea statements, and collapsing similar idea statements. This initial extraction activity resulted in 360 idea statements (See Appendix H). My next task was to categorize the 360 idea statements that emerged from the initial extraction activity, using the approach described by Morrison (2003) as emerging themes. After this analytical step, the Round 1 items were grouped into eight themes or categories, as follows: • Constructivism and Instructional Design • Constructivism and Social/Community/Culture Interplay • Constructivism and Learning Environment • Constructivism and the Instructor • Constructivism and the Learner • Constructivism Learning Activities • Constructivism and Assessment • Constructivism as Theory In addition, within each of the eight themes, a number of sub-themes emerged. These themes and sub-themes were used to organize the Round 1 data. To ensure that I had interpreted idea statements and placed them in appropriate themes and sub themes, I consulted with a colleague well versed in constructivism. She confirmed my themes and the placement of the idea statements within them. In reviewing the results of Round 1 data summary, I determined that such a large number of ideas would result in poor quality discussion among the participant/experts in 102 Round 2. In order to make the process more manageable, before beginning the new Round, I divided the 38 participant/experts into two groups — 2A and 2B, with random assignment to each group. This required that I sub-divide the full list of the idea statements generated and separate them by originator, thus allowing each group to discuss only the points generated by their members. The result was two lists of idea statements: 2A contained 192 idea statements and 2B contained 168 idea statements, each under the original eight themes (see Tables 9 and 10). As is shown by the sheer size of the tables and the list from both groups 2A and 2B, the topics were dense and difficult to wade through. In order to facilitate the upcoming discussion, I grouped five of the main themes (instructional design, social/community, environment, instructors, and learners) into further emergent sub-themes (such as design context, design tasks, pedagogy, activities, atmosphere, structure, presence, assessment, challenges, to name a few). Three of the main themes did not lend themselves to further division. Basically, the data gathered was too dense to be of any use to the participant-experts. The tables below, while interesting in the variety of idea statements, are too unwieldy to pull out individual items of interest. Table 9 Round 2A Idea Statements KJnstrMionalDesign , • '* • ' , ' \ '" , • - V - > 1 Design Context 1. Authentic tasks/problems (estudy29) 2. Communities of practice (estudy29) 3. Learning as social and dialogical process (estudy29) 4. Opportunities for exchange between experts and apprentices (estudy29) 5. Knowledge community (estudy29) 6. Learner rather than teacher centred (estudy07) 7. Multiple perspectives (estudy31) 103 Design Tasks 8. Information gaps stimulate learners to explore, analyse and synthesise (estudy08) 9. Must address learners attitude and perception towards learning (estudy08) 10. Collaborative tasks where students work together, share perspectives, and contribute to solutions (estudy08) 11. Learning supports provided so learner's motivation is maintained and time is somewhat managed (estudy03) 12. Design media to support dual coding (estudy08) 13. Visual representation of ideas e.g. collaborative concept map (estudy08) 14. Arrangement of tasks from simple to complex in line with Bloom's taxonomy (estudy08) 15. Use granularity in tasks? (estudy08) 16. Use cognitive task analysis? (estudy08) Pedagogy 17. As learners become more self-directed, the objectives may be negotiated in some areas or may not be achieved in the same order as one might have planned as a designer (estudy03) Activities 18. Feature different types of activities to accommodate a range of learning styles (estudy07) 19. Individual tasks with opportunities to see how others solved them, followed by discussion and reflection (estudy08) Social/Community /Culture Social Aspects 20. Learning is a product of negotiation with others (estudy25) (estudy03) (estudy04) (estudy05) (estudy02) 21. Meaning is constructed by engagement and interaction with the world/issues/context (estudy03) (estudy04) 22. Meaning is constructed with other people (estudy03) (estudy04) 23. Interaction, collaboration, challenge & feedback from others is key to helping students to establish accuracy & application of their ideas (estudy05) (estudy02) (estudy04) 24. Social aspects of learning can shape the nature and quality of learning (estudy04) 25. Our existing understandings or mental constructs (i.e. prior knowledge) may be challenged by the mental constructs of others (estudy02) 26. People *can* learn without direct cultural/social mediation (e.g., independent study, not that they can escape their prior learning and influences of culture) but that we all learn more easily in social contexts where we can seek assistance (the idea of ZPD) and verify our thinking with others (estudy26) Community and 27. The processing of knowing is mediated by community and culture and the tools of Culture that community and culture (especially language) (estudy26) 28. Knowledge is constructed by individuals actively in order to give meaning to socially accepted and shared notions and that an individual's construction of knowledge is tentative (estudy26) Group Work/ 29. Social negotiation of meaning thru asynchronous/synchronous collaborative tools Collaboration (estudy02) Environment-Pedagogy 30. Critically merge own vision with disciplinary insight and experience (estudy04) 31. Accommodate diversity (estudy07) (estudy09) 32. Create realistic expectations? (estudy07) 33. Act as a guide/coach/facilitator without being too prescriptive or unstructured (estudy07) (estudy09) (estudy04) 34. Be responsive to learners needs as they arise (estudy07) 104 35. Challenges and scaffolds so that their students can experience phenomena (estudy31) Structure 36. Structure is appropriate - but it needs elasticity (estudy09) 37. Use tools (software, web applications) to provide guidance (estudyl3) 38. Have tools to encourage critical thinking and active feedback (estudy08) (estudy04) 39. Adapt the learning system to the learner's different levels of abilities or preferences (estudy08) Atmosphere 40. Create and maintain an effective and safe environment (estudy02) (estudy09) 41. Allows for thinking/reflection instead of just listening (estudy36) 42. Interesting and engaging (estudy31) 43. Motivational (estudy36) 44. Co-operative (estudy36) 45. Ethnically and culturally diverse (estudy09) 46. Essentially encourages/places value on the development of personal understanding/knowledge construction by the learner? (estudy 12) (estudy09) 47. Students encouraged to represent their ideas in multiple modalities? and from multiple perspectives (estudy31) Activities 48. Activities/experiences such as problem-based learning approaches, collaborative learning opportunities, open-ended learning opportunities, self-paced learning opportunities, and varying pathways through the subject matter (estudy36) (estudy 12) 49. Learning activities that stretches students to learn more but does not overwhelm (estudy05) (estudy08) 50. Provide good set of resources (estudy26) (estudy08) (estudy09) 51. Create and manage or implement media-rich opportunity for experiences (estudy09) (estudy04) 52. Active or higher-level thinking (i.e., increased effort) (estudy36) 53. Individual/independent content/concept mapping (estudy36) 54. High levels of purposeful activity, and often toward authentic goals decided by learners or negotiated with them (estudy 12) (estudy31) (estudy08) 55. Tasks should be based in a realistic mix between knowledge of the world and knowledge of the discipline (estudy31) 56. Opportunities for micro and macro exploring either as individual or group (estudy09) 57. Prior learning is key - setting upon which and the tools by which further learning occurs (estudyOl) 58. Active and meaningful interaction with the learner's real environment (Akhras and Self, 2000) (estudy08) Assessment 59. Promote a number of different ways to achieve/apply/assess the learning (estudy29) (estudy09) (estudy36) Features/ 60. Collaboration on tasks, shared interpretation and dialogue contribute to the Characteristics creation of a community of learners (estudy03) (estudy36) (estudy31) (estudy 12) 61. In-depth analysis (estudy36) 62. Inquisition or investigation (estudy36) 63. Research-oriented (estudy36) 64. Sensitive to context and its importance in influencing learning (estudy08) 65. Emphasise active participation of learners with the content and with each other, with the intent of helping build a high degree of relevance to individuals and groups (estudy 12) 66. Multi-modal, rich, varied and multi-faceted (estudy37)(estudy09) 105 67. Is practical (estudy37) 68. Opportunities/facilitation abductive, multiloguing about the results of the exploration (estudy09) 69. People are always learning through various activities, educational and informal (estudy08) 70. The process is iterative and lifelong so that our own "meaning-making" is constantly changing (estudy02) 71. Progress from one stage of development to another over time (Vygostky's zone of proximal development) (Akhras and Self, 2000) (estudy08) Instructors • • . , .. - ,.' _• . Leadership 72. Step aside and allow the class to figure out how it will go down the path that is established (estudy02) Preparation 73. Responsible for the course resources (including physical or human - peer, technical support) (estudy05) (estudy08) 74. Select the major learning outcomes that the instruction is supposed to foster (estudy26) 75. Provide relevant examples (estudy02) 76. Learning activities value need to allow for reflection, interaction, critique and time to generate new learning/self-reflection (estudy03) (estudy37) 77. Ability to use the tools, processes and procedures(estudy04) Strategies 78. Provide opportunities not always answers (estudy09) 79. Avoid direct instruction (estudy37) 80. Watch for the 'teachable moment' and intervene (estudy07) (estudy26) 81. Support multiple conceptions and promote critical thinking that stretches boundaries (estudy02) (estudy05) (estudy08) 82. Learning strategies that allow for authentic learning and new experiences (estudy02) 83. Strategies for learners to work effectively in online learning groups? (estudy05) 84. Facilitate learners' active participation/dialogue/interaction (estudy03) (estudy29) (estudy05) 85. Flexible to respond to different starting points for different individual learners (estudy37) (estudy08) (estudyOl) (estudy29) (estudy03) 86. Provide opportunities for learners to have control and choice within the parameters of the course (estudy03) 87. Flexible in judging outcomes/success of learning (estudy37) Presence 88. Have a good online presence (estudy05) 89. Foster a positive/safe attitude/environment (estudy05) (estudy02) 90. Authentic help to assist students achieve their goals? (estudyl3) (estudy31) (estudy08) (estudy03) 91. Open, encourage and maintain clear lines of communication/dialogue (estudy37) 92. Kind (estudyl6) (estudy02) 93. Value individual experience (estudy07) Challenges 94. Personalize the online classes (estudy05) 95. Lack of training before teaching online (estudy05) 96. Need to understand the 'science of mind' before they can practice the art of good teaching or instructional design (estudy07) 97. Need to learn the pedagogy of their discipline (estudy07) Titles/Descriptors 98. Coach, Guide and facilitator? (estudy09) (estudy07) (estudylO) (estudy03)(estudyl3) (estudy36) (estudyl2) (estudy29) (estudy26) (estudy31) 99. Mediator, monitor (estudy36) (estudy37) (estudy08) 100. Knowledge expert/resource who can provide different levels of knowledge and 106 expertise for undergraduate and graduate students (estudy05) (estudy03) (estudy 13) (estudy36) (estudy29) (estudy26) 101. Consultant (estudy 13) (estudy36) 102. Moderator (estudy07) (estudy36) 103. Learning environment designer (estudy 12) 104. Model for effective strategies (estudy03)(estudy29) 105. Advisor (estudy 13) (estudy36) 106. Manager/project manager (estudy36) 107. Learning agent (estudy36) 108. Critical friend (estudyOl) 109. Learning Architect (estudy36) 110. Mentor (estudy36) 111. ?NetworkingNode?(estudy36) 112. Liaison (estudy36) 113. "Devil's advocate"? (estudy36) 114. Counsellor (estudy36) 115. Director (estudy36) hLeamer^ '/VVtf*",\ti>'>;hi?'>"';V'r • 'VJ' i ^'; ,^Vv,l;''^  Prior Experience/ 116. Share/draw on relevant work & research experiences/knowledge? (estudy05) Knowledge 117. Learning built on prior knowledge (estudy07) (estudy37) (estudy26) 118. Requires a degree of self-knowledge and awareness on part of learners (estudy37) 119. Have some understanding of constructivist methodology in order to develop appropriate skills and become successful as more active, self-reflective learners? (estudy29) Responsibility 120. Opportunity to design and carry out one's own learning agenda (estudy37) 121. Take responsibility for own knowledge construction (estudyl2) (estudy03) 122. Develop an understanding of possible solutions or responses, as they work individually or collaboratively (estudyl2) (estudy02) 123. Share information and growing knowledge with colleagues (estudyl2) 124. Take steps towards thinking and acting appropriately in the discipline or skill area (estudy03) 125. Learners are active as individuals and as group members (estudy37) (estudy04) (estudy 12) 126. Seeks meaning through constructive processes - forming, elaborating, and testing mental structures until one makes sense to him/her (estudy03) (estudy07) 127. Knowledge is not transferred, but is constructed by the learners (knowers) (estudy04) (estudyOl)(estudy03) (estudy05) (estudy07) 128. Learning takes time and requires motivation (estudyl3) 129. Influenced by teaching and perceived expectations as well as by various aspects of the social context it takes place within (estudy07) 130. Be considered in the context of authentic tasks, communities of practice and engaged in learning as a social and dialogical process (estudy29) 131. Messy and complex and can be approached from multiple perspectives (estudy26) Instructor 132. Encourage ownership/proaction in learning (estudy03) (estudy02) 133. Regular contact with the learners (estudy04) (estudy37) 134. Work guided by learners needs (estudy04) 135. Support learners by modelling the role of a problem solver (estudy07) 136. Establish trusting relationships with students (estudy02) 107 137. Use scaffolding techniques that support the learner (estudy26) (estudy08) 138. Encourage maturity through helping students be more self-directed and self-regulated (metacognition) in their learning (estudy05) 139. Offer appropriate/timely feedback (estudy05) 140. Encourage students to share, discuss, and (critically) reflect on what they are learning (estudy26) (estudy08) (estudyOl) 141. Works with the learners to establish criteria for evaluation of their learning (estudyl2) 142. Skilled at enabling students (estudy37) 143. Flexible as different students will move in different ways and at different speeds (estudy37) 144. Encourage learners to assess their work themselves and to present to other learners to obtain feedback(estudy08) 145. Manage group dynamics so that collaborative work will be meaningful and useful without learners getting on each other's nerves or some learners continuing to be silent and some learners continuing to be domineering (estudy08) (estudy02) 146. Train student leaders to facilitate discussions (estudy08) (estudy02) 147. Assign roles and ensure learners take turns while discussing (estudy08) (estudy02) 148. Monitor and respond to ongoing concerns and needs of different students (estudy08) (estudyOl) 149. Challenges/assists learners to see links between and among multiple conceptions/perspectives (estudy03) (estudy26) (estudy31) (estudy08) (estudyOl) (estudy02) 150. Works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, self-actualized learners (estudyl2) (estudyOl) 151. Help the learner become more independent and successful as a learner generally (e.g., learn to manage their time, to learn how to access resources such as the library and develop research skills) (estudy03) (estudy 13) 152. Shows learners how they have moved forward (estudy 37) Activities ' , , 153. Experiential learning (estudy09) (estudy05) 154. Role-play scenarios where students research their roles? (estudy09) 155. Group/open ended discussions/dialogue (estudy04) (estudy07) (estudy09) (estudy26) (estudy36) 156. Learning-by-doing/activity based learning(estudy04) (estudy20) 157. WebQuests (estudy04) (estudy09) 158. Critical inquiry (estudy04) 159. Problem based/inquiry based learning (estudy04) (estudy09) (estudy03) (estudy26) (estudyOl) 160. Authentic learning tasks (estudy03) (estudy31) (estudy02) 161. Simulations (estudy05) (estudy07) (estudy26) (estudy36) (estudy02) 162. Reflective individual writing assignments (estudy05) 163. Evidence based practice? (estudy07) 164. Working models? (estudy07) 165. Interactive multimedia? (estudy07) 166. Collaborative construction of common projects or representations/ (estudy26) (estudy31) (estudyOl) (estudy02) 167. Small-group inquiry exercises, reported out to the larger class (estudy02) 168. Guided discovery learning approaches (estudy03) (estudy36) 169. Independent? reading, whether online or offline? framed as reading as self-guided RESEARCH (estudy02) 170. Case-based approach? (estudyOl) (estudy36) (estudy02) 171. E-portfolios (estudyl3) 172. Co-operative learning (estudy26) (estudy36) 108 173. Synchronous online sessions *if* they involve exchange and discussion (not online lectures) (estudy26) 174. Research papers and presentation of findings (estudy36) 175. Peer teaching (estudy36) 176. Content/topic development (estudy36) 177. Content/topic resource gathering (e.g., "scavenger hunt") (estudy36) 178. Gaming (estudy36) 179. Debates? (estudy02) ; Assessment • > :-180. Peer assessment (estudy07) 181. Teacher assessment of students (estudy07) 182. A l l learning activities, problems and assessments need to relate to intended objectives and outcomes (estudy03) 183. Learning can have aims but these cannot be too structured ie not competency or right/wrong assessment/evaluation techniques (estudy37) {••Theory '.''.,J.':,%;V 184. Descriptive theory or as a prescription for how to teach (estudy06) 185. Not *a* theory of learning but a descriptive term for a series of similar, but not always mutually compatible, theories of learning (estudy26) 186. Theory based on an assortment of academic disciplines and ideas that has been formalized in the educational area (estudy05) 187. Based on a weak subjectivist epistemology, in which the focus is on how the individual internally? constructs meaning. A possible but not necessary consequence of premising learning on subjectivism poor rigour or mechanisms for evaluating learning? Thus the criticism that constructivism can lead to 'content-free learning' where the focus is on feelings (estudyOl) 188. Differences within the schools of constructivism about the importance of social interaction versus individual cognition and our ability to know what is real (estudy02) 189. Assumes a subjectivist view of the world, even a post-modern one of multiple realities and views and no "grand narratives" (estudy02) 190. Theory of real learning mostly centred on activities outside of courses, classrooms and teachers (estudy 13) 191. One risk is that it can lean toward relativism: anyone can build his/her own knowledge, but there is no "truth". Social version: we agree on some knowledge construct, disregarding its accuracy (estudy04) 192. The process is subjective and cognitive and recognizes that knowledge is ambiguous and truth uncertain (estudy02) 109 Table 10 Round 2B Idea Statements j Instructional Design« • , Design Context 1. Authentic tasks/problems (estudy20) 2. Involvement in meaningful contexts is critical for student understanding (estudy20) 3. Construction, maintenance and facilitation of a true S O C I A L C O N T E X T is perhaps the biggest challenge in online instruction, especially when the learners are culturally diverse? (estudy 18) Design Tasks 4. Requires feedback loops regarding the learning process which likely includes some metacognition (estudy 10) (estudy06) 5. Collaborative tasks where students work together, share perspectives, and contribute to solutions (estudy06) (estudy23) 6. Discussion to have some mechanism for facilitation and some "rules of engagement"? (ideas regarding process, netiquette, etc.) (estudy 10) Pedagogy 7. Scaffolding of learning onto prior experiences(estudy22) 8. Scaffold defined processes of inquiry, knowledge construction (estudy06) 9. Challenges to current thinking, fol lowed by opportunities to build alternative understandings (estudy06) (estudy23) Activit ies 10. Requires reflection on what has been learned/personal goals/needs (estudy 10) (estudy06) 11. Meaningful problem-solving activities (estudy06) 12. Finding activities that are designed at the right level of challenge, w i l l stimulate curiosity and motivation (estudy 14) 13. Individual tasks with opportunities to see how others solved them, fol lowed by discussion and reflection (estudy06) 14. Guided activities where the content is shown to be useful in addressing problems and tasks (estudy06) Social Aspects 15. Learning is a product of negotiation with others (estudy25) (estudy20) (estudy34) (estudy21) 16. Meaning is constructed by engagement and interaction with the world/issues/context (estudy06) (estudy20) (estudy34) 17. Meaning is constructed with other people (estudy06) (estudy20) (estudy34) 18. Social ly mediated process (estudy22) (estudy23) 19. Interaction, collaboration, challenge & feedback from others is key to helping students to establish accuracy & application of their ideas (estudy34) (estudy23) (estudy 14) (estudy06) 20. Social aspects o f learning can shape the nature and quality of learning (estudyl8) (estudy23) 21. Our existing understandings or mental constructs (i.e. prior knowledge) may be challenged by the mental constructs of others (estudy23) 22. Sharing multiple perspectives and hearing the constructions of others allows one to question our original assumptions (estudy23) Community and 23. Highly cultural - we construct meanings in very different ways based upon belief Culture systems, societal and community group values. Even behaviourist systems of education are structured for the cultural in which they are situated, and the experience one has in such systems impact the construction of knowledge (estudy 16) 24. A safe environment that allows for diversity and inclusively otherwise one does 110 not construct their own meanings - they may their negate their own worldview in favour of the dominant one (estudy 16) Group Work/ 25. Placing students in groups and encouraging students to 'own what they know' has Collaboration become much more typically used as an educational tool in the constructivist era (estudy 15) 26. Many put students into groups because they have less to grade as product from group papers (estudy 15) 27. Online courses, if they use the technologies well, can provide opportunity for learning within a communal group (estudy21) 28. Social negotiation of meaning thru asynchronous/synchronous collaborative tools (estudy24) 29. Provide forums that are private for groups of 3-4 (estudy 17) Environment Pedagogy 30. Act as a guide/coach/facihtator without being too prescriptive or unstructured (estudy06) 31. Consider the maturity of the students (estudy25) 32. Respect the autonomy of the individuals involved as they engage in a negotiation of meaning with others (estudy25) Structure 33. Structure is appropriate - but it needs elasticity (estudy21) 34. Provide guidelines to success (estudy 15) 35. Have tools to encourage critical thinking and active feedback (estudy06) 36. Encourage development of new tools and resources (estudy06) Atmosphere 37. Create and maintain an effective and safe environment (estudylO) (estudyl4) 38. Allows for thinking/reflection instead of just listening (estudy 17) (estudy 17) (estudy 16) 39. Learner-centered (estudy23) 40. Essentially encourages/places value on the development of personal understanding/knowledge construction by the learner (estudy22) (estudy24) (estudy23) (estudy24) (estudy06) (estudy 16) (estudylO) 41. An understanding that there are multiple ways that 'knowledge' is absorbed and constructed (estudy 18) Activities 42. Activities/experiences such as problem-based learning approaches, collaborative learning opportunities, open-ended learning opportunities, self-paced learning opportunities, and varying pathways through the subject matter (estudy21) 43. Learning activities that stretches students to learn more but does not overwhelm (estudylO) 44. Provide good set of resources (estudy 14) (estudy22) (estudy06) 45. Create and manage or implement media-rich opportunity for experiences (estudy21) (estudy35) (estudy06) 46. High levels of purposeful activity, and often toward authentic goals decided by learners or negotiated with them (estudy25) 47. Prior learning is key - setting upon which and the tools by which further learning occurs (estudy 15) Assessment 48. Monitor and assess student learning via authentic assessments, that is, projects and problem-solving tasks resembling those encountered outside of the classroom (estudy06) 49. Promote a number of different ways to achieve/apply/assess the learning (estudy25) 111 50. Collaboration on tasks, shared interpretation and dialogue contribute to the creation of a community of learners (estudy23) (estudy2l) (estudy25) (estudylO) (estudy06) 51. Self-experiential referencing (i.e., relevant and meaningful) (estudyl6) 52. Sensitive to context and its importance in influencing learning (estudy25) 53. Emphasise active participation of learners with the content and with each other, with the intent of helping build a high degree of relevance to individuals and groups (estudy25) (estudylO) 54. Multi-modal, rich, varied and multi-faceted (estudy25) (estudy22) (estudy06) 55. Bransford, Brown, & Cocking conclude that learning *environments* should be: Student-centric; Knowledge-centric; Assessment-centric; Community-centric. (estudy33) 56. Socially-mediated (estudylO) 57. People are always learning through various activities, educational and informal (estudy06) ! Instructors -^ ' • •^•- - •v : * .^.UV"^ •'"->'. * 'i Leadership 58. Lead and follow (estudy 15) 59. Step aside and allow the class to figure out how it will go down the path that is established (estudy 15) (estudy20) (estudy23) Preparation 60. Lay down the groundwork/provide the frame of reference (estudy 15) 61. Create appropriate constructivist assignments (estudy22) 62. Responsible for the course resources (including physical or human - peer, technical support) (estudylO) (estudy24) 63. Provide relevant examples (estudy 16) 64. Space for dialogue about social and contextual issues, and even chit-chat, as well as 'on-task' interaction(estudyl 8) 65. Learning activities value need to allow for reflection, interaction, critique and time to generate new learning/self-reflection (estudy25) (estudy23) (estudy24) Strategies 66. Not primarily a transmitter of information(estudy21) 67. Avoid direct instruction (estudy24) 68. Support multiple conceptions and promote critical thinking that stretches boundaries(estudy23) (estudy34) (estudy 16) (estudy 18) (estudylO) 69. Strategies for learners to work effectively in online learning groups (estudy20) (estudy 17) (estudy 16) 70. Facilitate learners' active participation/dialogue/interaction (estudy34) (estudyl4) (estudy 16) (estudylO) 71. Provide opportunities for learners to have control and choice within the parameters of the course (estudy23) Presence 72. Foster a positive/safe attitude/environment (estudy 14) (estudy 17) 73. Responsible for the social mileu (estudylO) (estudy20) 74. Open, encourage and maintain clear lines of communication/dialogue (estudy06) (estudy23) (estudy 16) (estudy 17) 75. Uses proactive intervention (with a spirit of caring vigilance) (estudylO) 76. Kind (estudy 16) 77. Encourage the social construction of knowledge (estudy23) 78. Encourage voices not only from the dominant culture (estudy 16) 79. Responsive (estudy 18) Challenges 80. Need to learn the pedagogy of their discipline (estudy06) 81. Need to relinquish power and certainty (estudy25) 82. Need do a deep values inventory on themselves (estudy25) 83. Need to put some of our own most treasured orthodoxies on hold (estudy25) 84. Need to integrate the cognitive (thinking skills) and the affective (e.g., motivation) 112 Features/ Characteristics to be effective (estudy 10) 85. To weave a delicate thread through personal experiences/perspectives, course content, theory, and discussion (estudy34) 86. Shift to learner-centered philosophy (estudy23) Titles/Descriptors 87. Coach, Guide and facilitator (estudy22) (estudy06) (estudy25) (estudy 10) (estudy34) (estudy20) (estudy33) (estudy26) (estudy 16) 88. Mediator, monitor (estudy24) (estudy06) 89. Knowledge expert/resource who can provide different levels of knowledge and expertise for undergraduate and graduate students (estudy06) (estudy26) (estudy23) 90. Consultant (estudy 17) 91. Cheerleader (estudy25) 92. Negotiator (estudy25) 93. Team leader(estudy25) 94. Provocative questioner (estudy25) (estudy 14) 95. Learning environment designer(estudy25) 96. Co-Learner (estudy 10) (estudy20) (estudy23) 97. Model for effective strategies (estudy20) (estudy 14) 98. Manager/project manager (estudy33) 99. Critical friend (estudy33) Learner ' • . , * - / Prior Experience/ 100. The extent to which an educator will provide a recipe of guidelines for their Knowledge students often depends on the academic level or experience of the students they are dealing with (estudy22) 101. Share/draw on relevant work and research experiences/knowledge (estudy35) 102. Learning built on prior knowledge (estudy34) (estudy20) (estudy23) (estudy24) (estudy 10) (estudy35) (estudy26) 103. Learners individually interpret their experiences and build their own internal representations of knowledge (estudy24) . Responsibility 104. Opportunity to design and carry out one's own learning agenda (estudy 10) 105. Take responsibility for own knowledge construction (estudy20) (estudy25) 106. Develop an understanding of possible solutions or responses, as they work individually or collaboratively(estudy20) (estudy25) 107. Share information and growing knowledge with colleagues(estudy20) 108. Engaged in critical thinking/reflection (estudy23) (estudy20) (estudy 18) 109. Learning is intentional, personal, and useful for the learner (estudy20) 110. Learners are active as individuals and as group members (estudy20) (estudy25) (estudy 10) (estudy24) (estudy 18) 111. Seeks meaning through constructive processes - forming, elaborating, and testing mental structures until one makes sense to him/her (estudy21) (estudy25) (estudy06) (estudy 18) 112. Knowledge is not transferred, but is constructed by the learners (knowers) (estudy25) (estudy 18) (estudy21) (estudy06) 113. Learning takes time and requires motivation (estudylO) 114. Learning is a social process (estudyl4) (estudy20) 115. An understanding that learning and new understandings can develop using various tools and approaches - applied and theoretical (estudy 18) 116. Be considered in the context of authentic tasks, communities of practice and engaged in learning as a social and dialogical process (estudy 14) Instructor 117. Encourage ownership/proaction in learning (estudy21) (estudy 17) 118. Self-regulation of the learning environment (estudy21) 119. Work guided by learners needs (estudy06) 120. Support learners by modelling the role of a problem solver (estudy24) 121. Arrange online instruction to meet individual learner needs (estudy21) 113 122. Establish trusting relationships with students (estudy06) (estudy 16) 123. Support students to negotiate technical and performance requirements (estudy06) 124. Use scaffolding techniques that support the learner (estudy24) (estudy20) (estudy 14) 125. Offer appropriate/timely feedback (estudy05) (estudy 16) 126. Encourage students to share, discuss, and (critically) reflect on what they are learning (estudy20) (estudy23) 127. Support students diverse levels of knowledge (estudy 14) (estudy23) 128. Offer opportunities for students to follow their own interests (estudyl4) 129. Monitor and respond to ongoing concerns and needs of different students (estudy06) (estudy35) 130. Challenges/assists learners to see links between and among multiple conceptions/perspectives (estudy23) (estudy34) (estudy20) (estudy26) 131. Respect that learners will create valuable knowledge (estudy25) 132. Works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, self-actualized learners (estudy25) 133. Allows sufficient time within the course for students to work collaboratively (estudy20) ^Activities 134. Group/open ended discussions/dialogue (estudylO) (estudy34) (estudy23) 135. Learning-by-doing/activity based learning(estudy04) (estudy20) 136. WebQuests (estudy 14) 137. Critical inquiry (estudy04) (estudy34) (estudy20) 138. Problem based/inquiry based learning (estudy24) (estudy22) (estudy 18) (estudy20) (estudy 14) (estudy34) 139. Authentic intellectual work (estudy24) 140. Motivating assignments/tasks(estudy24) 141. Simulations (estudy34) 142. Interactive multimedia (estudy34) 143. Collaborative construction of common projects or representations (estudy06) (estudy20) (estudy34) 144. Discussion and reflection around team activities(estudy06) 145. Threaded discussion of assigned content or study questions(estudy06) 146. Guided discovery learning approaches (estudy35) 147. Independent? reading, whether online or offline framed as reading as self-guided RESEARCH (estudy 18) 148. Case-based approach (estudy 18) (estudy34) (estudy 14) 149. Project-based learning (estudy20) (estudy 14) (estudy34) 150. Resource-based learning (estudy20) 151. Co-operative learning (estudy20) (estudy 14) 152. Computer mediated communication (estudyl4) 153. Self-directed learning? (estudy34) ^Assessment = ^ ^ " ' "• 154. Peer assessment (estudy 15) (estudy 10) 155. Student self assessment (estudy24) (estudyl5) 156. Portfolio assessment(estudy24) 157. Include methods for demonstrating learner skill/competence/depth of understanding (estudylO) 158. Activities in which learners engage should be learner centered and measured against outcomes (estudy23) Theory -> W- ~': • • >" ^ : , 159. 'Common sense' theory of learning (estudy 15) 160. Descriptive theory or as a prescription for how to teach (estudy06) 114 161. Not a "theory of learning" at all, but rather a philosophy of learning (or more accurately, an epistemological perspective) (estudylO) 162. Knowledge does not exist independent of the learner (estudy21) 163. Belief in multiple realities and multiple perspectives (estudy21) 164. It is important to not relegate constructivism to a single theory - it takes many forms and is based in many philosophical stances (estudy21) 165. Knowledge is unique to the individual, and it is created by the individual in a peculiar way, so it calls into question whether any two learners can share the same knowledge (estudy25) 166. Distinction between cognitive constructivism, where learners experience some kind of dissonance, and engage in cognitive activities to make sense of the world, and socio-cultural constructivism, which emphasizes that learning is embedded in social and cultural practices (estudy25) 167. A philosophy that draws on constructivist tenets embraces the idea that learning is created and learners may take away more from the experience that originally anticipated (estudy23) 168. There is no 'reality', there are multiple realities and no right or wrong. Learning is constructed as a result of the process/es where multiple perspectives are revealed, discussed, then reconstructed (estudy34) (estudy21) Round 2 Data In Round 2 opening participation and results were inadequate in terms of generating any useable data. While 24 participant/experts contributed to Round 2A (10 participant/experts for a total of 95 postings) and 2B (14 participant/experts for a total of 112 postings), many indicated that the task was overwhelming because of the number of idea statements to be dealt with. On reviewing the task set for participant/experts and the size of the idea statement sets which each group had to deal with, I decided to reduce the idea statement sets and group sizes to provide more manageable interaction. In doing so, I closed Round 2A and 2B and initiated Round 2.2. In Round 2.2 participant/experts were randomly assigned to one of eight private spaces which were assigned the task of discussing idea statements within just one theme. This grouping was further divided when I discovered that three of the themes held more than the intended maximum of 30-35 items. The final result was the creation of eleven small groups (2A, 2B, 2C, 2D, 2E, 2F, 2G, 2H, 21, 2J and 2K), consisting of three to four participant/experts 115 with a maximum of thirty-four items per group to explore. In all, 29 participant/experts participated in Round 2.2. Table 11 displays a breakdown of participation by group in Round 2.2. Table 11 Breakdown of Participation in Round 2.2 Sub- % Group Dates Number of Round Engagement Postings 2.2 A 33% May 14-June 6 12 2.2B 75% May 14-June 4 9 2.2C 50% May 14-June 2 24 2.2D 75% May 14-June 5 10 2.2E 75% May 14-June 4 16 2.2F 75% May 14-June 7 . 12 2.2G 100% May 14-June 6 10 2.2H 100% May 14-June 7 9 2.21 100% May 14-June 7 11 2.2J 100% May 14-June 7 8 2.2K 66% May 14-June 6 6 As can be seen in the table, four of the groups have full participation while only one group found itself with a minimum of interaction. Round 2.2 resulted in the deletion and combination of many of the original idea statements, and in all 51 idea statements emerged from this Round, as presented in Table 12. These idea statements now became more manageable for the participant-experts. 116 Table 12 Idea Statements Generated by Round 2.2 1. Encourages/values the development of personal understanding/knowledge construction by the learner. 2. The core of the design is learner-centred. 3. The process is iterative and lifelong so that our own "meaning-making" is constantly changing. 4. Understanding that there are multiple ways that 'knowledge' is absorbed and constructed. 5. Supports multiple conceptions and promote critical thinking that stretches boundaries. 6. Fosters active/higher-level thinking and reflection. 7. Emphasise/facilitate active participation of learners with the content (dialogue) and with each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group. 8. Learners are encouraged to seek knowledge and experience from different types of sources and perspectives. 9. Sharing multiple perspectives and hearing the constructions of others allows one to question our original assumptions and to develop a shared meaning. 10. Learning is constructed as a result of the process(es) where multiple perspectives are revealed, discussed, then reconstructed. 11. Provides opportunities, but not always answers. 12. Offers opportunities for students to follow their own interests. 13. Learners should have opportunities to solve complex, realistic problems similar to those that the learners themselves would encounter outside the classroom. 14. Uses a learner-centred approach to teaching, which involves sharing authority with learners. 15. Works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, self-actualised learners. 16. Encourages the voices of all learners and is respectful of what the learner can contribute to the learning enterprise. 17. Assessment should avoid overly narrow measurement of discrete skills or knowledge without a more complete context of interpretation. 18. Open-ended discussions based on critical inquiry/authentic intellectual work. 19. Builds skills and abilities while working on authentic tasks and problems. 20. Provides and balances self-directed and collaborative learning opportunities. 21. Seeks engagement with a community of learners or practice. 22. Becomes a metacognitive coach for learners. 23. The instructor is flexible enough to support student expressions of different levels of knowledge. 24. Should create an environment with a social context. 25. Assessment should often include learners' reflections on their goals and accomplishments. 26. Builds on prior learning (knowledge) of individuals. 27. Critically merges the learner's knowledge of the world with disciplinary insight and experience. 28. Emphasises experiential, authentic learning, which is decided on by learners or negotiated with them. 29. Encourages exploration of multiple paths and forms and facilitates opportunities for varying experiences and pathways through the subject matter. 30. Social negotiation of meaning can happen through asynchronous / synchronous collaborative tools. 31. Assessments should accommodate the needs of diverse learners. 32. Learners may take away more from the experience than originally anticipated. 33. Support the goals of learners by facilitating appropriate forms of learner collaboration and discussion. 34. Flexible in responding to different starting points for different individual learners. 35. Learning activities are "authentic" and require reflection. 36. Knowing is mediated by community/culture and the tools of that community and culture, so be aware of and exploit cultural differences. 37. Provide activities at the right level of challenge for learners. 38. Should accommodate a shift in expected learning outcomes. 39. Provides opportunities for dialogue between novices and experts. 40. Assessment systems should examine a learner's full range of knowledge, skill, and attitudes toward a 117 domain of expertise. 41. Learners should haveaccess to assessment criteria and methods, and be able to negotiate at least some of the terms of the assessment. 42. Knowledge does not exist independent of the learner. 43. Learners interact with a media-rich and resource-rich environment in building knowledge. 44. Provides a safe environment, encouraging risk-taking and authentic dialogue. 45. Emphasizes co-operative learning (collaborative construction of common projects or representations). 46. Constructivism assumes a subjectivist view of the world, even a post-modern one of multiple realities and no "grand narratives." 47. Provide a variety of ways of introducing course material so those students with different learning styles are accommodated. 48. Provides a clear description of the rules of the course so that all learners understand the playing field. 49. Steps aside and allow the class to figure out how it will go down the path that is established. 50. Practised in the art of evidence-based teaching and instructional design. 51. A variety of learner performances and products (both individual and group-based) may be suitable for assessment, including: a) Content/topic development/resource gathering b) Creative works c) Problem-based learning d) Case-based learning e) Asynchronous discussion f) Portfolios/e-portfolios or collections of work samples g) Guided-discovery learning h) Evidence-based practice i) Webquests j) Independent reading framed as self-guided research k) Live performances 1) Inquiry reports m) Performance assessments and skill demonstrations n) Small group inquiry exercises, reported out to the whole class o) Research papers and presentation of findings p) Peer teaching q) Simulations r) Role-playing (gaming) s) Working models t) Synchronous on-line sessions Round 3 Data The purpose of this round was to give participant/experts the opportunity to evaluate and voice their opinions on the individual idea statements, which had emerged to this point. This was accomplished through a vote via an online survey. 118 Data Analysis Layout In all 33 participant-experts began the survey, and it was fully completed by 31. The data that resulted from the vote are displayed in three sets of tables: Question 1 -Tables 13 (13.1-13.7), Question 2 - Tables 14 (14.1-14.7), and Question 3 - Tables 15 (15.1-15.7). These tables are interwoven to facilitate discussion of commonalities and disparities across the questions/activities in this voting Round. Tables 14 and 15 are ordered, in terms of idea statements, in relation to the consensus levels on Tables 13 (the agreement/disagreement as to the idea statement representing a core defining element or feature of constructivism). The data from Question 1 (agree/disagree scale) garnered some interesting results. From the 51 idea statements created by previous Rounds and facilitator culling, total agreement (100% consensus) was reached on two idea statements. Consensus was reached on an additional 30 idea statements (90-99% consensus on 10 idea statements; 80-89% consensus on 20 idea statements). Just 16 idea statements failed to reach consensus level on this scale (70-79% consensus on 12 idea statements; 60-69% on four idea statements). Less than 50% consensus applied to just two idea statements. One remaining idea statement incorporated 20 classroom learning activities that reflected constructivist theory: of these only one activity attained the 80% consensus level set for this study (see Tables 13.1-13.7). This resulted in a total of 33 idea statements that received consensus. The data from Question 2 involved participant-experts, in response to the 51 idea statements, ranking them on a five point Likert-type scale, equivalent to one (1) essential; two (2) very important; three (3) important; four (4) somewhat important; five (5) not 119 important. The purpose of this vote was to establish shades of importance of each idea statement in terms of being a defining element of constructivism. This voting scale was completed by 31 to 33 participant-experts (a few idea statements were not ranked by all participant-experts) taking part in this Round. Consensus level on the 51 idea statements were for the most part aligned with the results of Question 1, with just two idea statements failing to achieve Round 1 90-99% consensus; six idea statements failing to achieve Round 1 80-89% consensus; three idea statements failing to achieve Round 1 70-79% consensus; and no idea statements failing to achieve Round 1 60-69% consensus (see Tables 14.1-14.7). The data from Question 3 involved participant-experts, in response to the 51 idea statements, ranking them on a two-point scale, indicating whether the idea statements reflected their personal view of constructivism, or a globally held view of constructivism. The voting scale was completed by 31 to 33 participant-experts (a few idea statements were not ranked by all participant-experts) taking part in this Round. Consensus level on the 51 idea statements were not closely associated with the results of Question 1 in most consensus categories. In fact, only in the 100%) category is there any consistency with the global view and the two idea statements achieving that level of consensus (see Tables 15.1 -15.7). Consensus 100% Question 1 Scale. Total agreement was attained on two idea statements (see Table 13.1). 120 Table 13.1 Results from Round 3 online vote - Question 1 (J 00%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree Encourages/values the development of personal understanding/knowledge construction 100% 0% by the learner. (N=33) The core of the design is learner-centred. (N=31) 100%, 0% Note: 1. The numbers within each statement vaiy: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. These two idea statements support much of the literature on constructivism and reflect current thinking in this learning theory. Many authors and researchers have noted that the tenets of constructivism encourage and value personal knowledge construction on the part of the learner, and at its core, constructivism should be learner-centred (Ally, 2004; Goldman & Torrisi-Steele, 2002; Delgarno, 2001; Doolittle, 1999; Murphy, 1997b; Wilson, 1997). What is noteworthy is the fact that these two idea statements achieved 100% consensus as representative of defining elements of constructivism, across all participant-experts in the study, indicating their acceptance as core to constructivist learning and teaching. Question 2 Scale. On the second scale, there was, in general, agreement with the vote on Question 1 (see Table 14.1). 121 Table 14.1 Results from Round 3 online vote - Question 2 (100%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Encourages/values the development of personal 64% 27% 9% 0% 0% understanding/knowledge construction by the learner. (N=33) The core of the design is learner-centred. (N=31) 42% 52% 3% 3% 0% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. .45 in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. The two statements which were ranked as defining ideas of constructivism by 100% of participant-experts on the two-point scale were, not surprisingly, ranked highly by participant-experts on this vote as well. Ninety-one percent of participants ranked the first statement as essential or very important, and ninety-four percent of participant-experts ranked the second statement as essential or very important. There was one ranking below the important category, which is worth noting. On the idea statement "the core of design is learner-centred" 3% (one participant-expert) relegated this idea statement to the relatively low somewhat important category. This position on the part of one participant-expert is anomalous with the first vote and with the literature, both of which indicate that learner-centeredness is a core feature of constructivism. Question 3 Scale. On the third scale, in which participant-experts were to indicate whether each idea statement reflected their personal view, or a globally held view of 122 constructivism, the data on the highest two idea statements were for the most part associated (to the consensus level) with the vote on Question 1 (see Table 15.1). It should be noted that the second idea statement "the core of design is learner-centred" was deemed to be a personal view, as opposed to a globally accepted view of constructivism, by 16% (five participant-experts). This vote against a global view indicates that these participant-experts do not see learner-centredness as a universally held belief in terms of constructivist learning and teaching. Nevertheless, both idea statements in this category did achieve consensus level. Table 15.1 Results from Round 3 online vote - Question 3 (100%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Encourages/values the development of personal understanding/knowledge 9% 91% construction by the learner. (N=33) The core of the design is learner-centred. (N=31) 16% 84% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Consensus 90-99% Question 1 Scale. Many of the 10 idea statements in the 90-99% consensus category on Question 1 reflect social constructivism (see Table 13.2). Concepts such as 'supports multiple conceptions', 'facilitates active participation', 'hear[s] the 123 constructions of others' are all noted in the literature on social constructivism as key to the construction of knowledge. Table 13.2 Results from Round 3 online vote - Question 1 (90-99%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree The process is iterative and lifelong so that our own "meaning-making" is constantly 97% 3% changing. (M=33) Understanding that there are multiple ways that 'knowledge' is absorbed and constructed. 97% 3% (N=33) Supports multiple conceptions and promote critical thinking that stretches boundaries. 97% 3% (N=32) Fosters active/higher-level thinking and reflection. (N=33) 94% 6% Emphasise/facilitate active participation of learners with the content (dialogue) and with 94% 6% each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group. (N=33) Learners are encouraged to seek knowledge and experience from different types of 91% 9% sources and perspectives. (N=33) Sharing multiple perspectives and hearing the constructions of others allows one to 91% 9% question our original assumptions and to develop a shared meaning. (N=33) Learning is constructed as a result of the process(es) where multiple perspectives are 91% 9% revealed, discussed, then reconstructed. (N=33) Provides opportunities, but not always answers. (N=32) 91% 9% Offers opportunities for students to follow their own interests. (N=32) 91% 9% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. The data indicate support for the notion of learning through interaction with others and with various knowledge sources, and the sharing of multiple perspectives. Items that reflect sharing multiple perspectives, multiple conceptions that expand knowledge boundaries, the many-faceted ways that knowledge and learning are constructed, and the active participation of learners with the content and with each other 124 all achieved a minimum of 91% consensus. The high level of agreement on the social elements of constructivist learning, of human interaction, and of valuing multiple perspectives are also consistent with the literature (Boudourides, 2003; Driscoll, 2000; Wenger & Snyder, 2000; Bruffee, 1999). Question 2 Scale. The idea statements in the 90-99% consensus category on Question 1 were ranked on the five-point essential - not important scale in Question 2 (see Table 14.2). A l l of the idea statements in this consensus category were deemed to be from important to essential by participant-experts. Two idea statements were ranked lower than others were however, with rankings of 12% each in the somewhat important - not important categories. Both of these idea statements contain the phrase "multiple perspectives" - i.e. sharing multiple perspectives, and where multiple perspectives are revealed Building on the work of Wilson (1997) I believe it is possible to be somewhat constructivist, or to believe in a constructivist approach to learning without adopting fully a postmodern belief system of multiple perspectives in relation to relative truth systems. I feel these statements and their ranking support this possibility. 125 Table 14.2 Results from Round 3 online vote - Question 2 (90-99%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank 1 (1-essential - 5-not important) 2 3 4 5 The process is iterative and lifelong so that our 33% 33% 30% 0% 3% own "meaning-making" is constantly changing. (N=33) Understanding that there are multiple ways that 33% 48% 18% 0% 0% 'knowledge' is absorbed and constructed. (N=33) Supports multiple conceptions and promote 41% 28% 22% 9% 0% critical thinking that stretches boundaries. (N=32) Fosters active/higher-level thinking and 33% 52% 15% 0% 0% reflection. (N=33) Emphasise/facilitate active participation of 45% 18% 33% 0% 3% learners with the content (dialogue) and with each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group. (N=33) Learners are encouraged to seek knowledge and 33% 36% 24% 6% 0% experience from different types of sources and perspectives. (N=33) Sharing multiple perspectives and hearing the 36% 27%o 24% 9% 3% constructions of others allows one to question our original assumptions and to develop a shared meaning. (N=33) Learning is constructed as a result of the 21% 30% 36% 9% 3% process(es) where multiple perspectives are revealed, discussed, then reconstructed. (N=33) Provides opportunities, but not always answers. 25% 31% 44% 0% 0% (N=32) Offers opportunities for students to follow their 28% 25% 38% 9% 0% own interests. (N=32) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Question 3 Scale. The idea statements in the 90-99% consensus category on Question 1 were ranked on the personal/global view scale also. Some items ranked low on the global view were consistent with those ranked lower on the Question 2 scale (see Table 15.2). 126 Table 15.2 Results from Round 3 online vote - Question 3 (90-99%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global The process is iterative and lifelong so that our own "meaning-making" is constantly 15% 85%> changing. (N=33) Understanding that there are multiple ways that 'knowledge' is absorbed and 24% 76% constructed. (N=33) Supports multiple conceptions and promote critical thinking that stretches boundaries. 28%> 72%> (N=32) Fosters active/higher-level thinking and reflection. (N=33) 24% 76% Emphasise/facilitate active participation of learners with the content (dialogue) and 27% 73% with each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group. (N=33) Learners are encouraged to seek knowledge and experience from different types of 15% 85% sources and perspectives. (N=33) Sharing multiple perspectives and hearing the constructions of others allows one to 36% 64% question our original assumptions and to develop a shared meaning. (N=33) Learning is constructed as a result of the process(es) where multiple perspectives are 39% 61% revealed, discussed, then reconstructed. (N=33) Provides opportunities, but not always answers. (N=32) 41% 59% Offers opportunities for students to follow their own interests. (N=32) 50% 50% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Of the nine idea statements in this consensus category, it is interesting to note, that only two attained the 80% consensus for this study as global belief statements. In terms of this study's established consensus level, most idea statements were deemed to represent participant-expert personal views, which is interesting, given that these same idea statements were deemed to represent core elements of constructivism on the first scale. Four idea statements were ranked very low as representing a global view: two of these mirror the ambivalent rankings in Question 2 - the "multiple perspectives" items 127 (64% and 61%). In addition, two items which incorporated ideas of "open opportunities for learners" were ranked low as global views (59% and 50%). These lower rankings are inconsistent with the literature on constructivism, which supports multiple perspectives and multiple modes of learning, and learner ownership and control (Moallem, 2001; Takala, Hawk, & Rammos, 2001; Duffy & Cunningham, 1996; Feltovich, Spiro, Coulson, & Feltovich, 1996). It is my speculation that in keeping with constructivist thinking, a number of participant-experts did not feel able to put a global view forward. One noted in discussions later in the final round that they were "...surprised by: — The relatively high incidence of "personal" ratings, over "global." In my ratings, I only marked 'personal' when I anticipated that my view would be a minority view. Consistently, the 'personal' ratings outnumbered the minority ratings—if that makes any sense at all" (Participant 6, July 26, 2004). Consensus 80-89% Question 1 Scale. There are 20 idea statements in this consensus category. Some of these idea statements reflect views about learner roles and activities, while others reflect possible direction for instructor roles - i.e. creating a learning environment with a social context (see Table 13.3). 128 Table 13.3 Results from Round 3 online vote - Question J (80-89%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree Learners should have opportunities to solve complex, realistic problems similar to those 88% 12% that the learners themselves would encounter outside the classroom. (N=33) Uses a learner-centred approach to teaching, which involves sharing authority with 87% 13% learners. (N=32) Works to nudge students toward the kind of intellectual responsibility they need to 87% 13% become self-motivated, self-actualised learners. (N=32) Encourages the voices of all learners and is respectful of what the learner can contribute 87% 13% to the learning enterprise. (N=31) Assessment should avoid overly narrow measurement of discrete skills or knowledge 87% 13% without a more complete context of interpretation. (N=31) Open-ended discussions based on critical inquiry/authentic intellectual work. (N=33) 85% 15% Builds skills and abilities while working on authentic tasks and problems. (N=33) 85% 15% Provides and balances self-directed and collaborative learning opportunities. (N=32) 84% 16% Seeks engagement with a community of learners or practice. (N=32) 84% 16% Becomes a metacognitive coach for learners. (N=32) 84%> 16% The instructor is flexible enough to support student expressions of different levels of 84% 16% knowledge. (N=31) Should create an environment with a social context. (N=31) 84%> 16% Assessment should often include learners' reflections on their goals and 84% 16% accomplishments. (N=31) Builds on prior learning (knowledge) of individuals. (N=33) 82%> 18% Critically merges the learner's knowledge of the world with disciplinary insight and 82% 18% experience. (N=33) Emphasises experiential, authentic learning, which is decided on by learners or 82%> 18% negotiated with them. (N=33) Encourages exploration of multiple paths and forms and facilitates opportunities for 81% 19% varying experiences and pathways through the subject matter. (N=32) Social negotiation of meaning can happen through asynchronous / synchronous 81% 19% collaborative tools. (N=32) Assessments should accommodate the needs of diverse learners. (N=31) 81% 19% Learners may take away more from the experience than originally anticipated. (N=31) 81% 19% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Authenticity in the area of student learning and assessment falls within this consensus category. The idea statements dealing with this issue cover a multitude of interest areas: learners should be provided with opportunities to solve authentic problems 129 that they would encounter outside the classroom. Furthermore, there is support for idea statements reflecting that discussion be based on authentic work, that disciplinary experiences be merged with learners' knowledge of the world, and that authentic learning activities be decided on by the learners themselves - or at least negotiated with them. Consensus on these idea statements is similar to that found in the literature on constructivism - in particular on authentic learning models (Luppicini, 2003; Perkins, 1999; Lave & Wenger, 1991). In this consensus category are a number of idea statements reflecting the need for the instructor to provide a suitable learning environment, while suggesting specific elements that lead to the provision of authentic learning activities. These include sharing authority with the learners, assisting learners in attaining responsibility for their own learning, and encouraging all learners while being respectful of learners' interactions and contributions. There is consensus on the role of instructors as metacognitive coaches for learners, and for flexibility in their own roles. With respect to assessment, it is agreed that it should accommodate the needs of diverse learners, be more flexible and less focused on the narrow measure of discrete skills, while encouraging learners to engage in reflection. Question 2 Scale. The idea statements in the 80-89% consensus category on Question 1 were ranked on the five-point essential - not important scale in Question 2 (see Table 14.3). 130 Table 14.3 Result from Round 3 online vote - Question 2 (80-89%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Learners should have opportunities to solve 24% 30% 30% 15% 0% complex, realistic problems similar to those that the learners themselves would encounter outside the classroom. (N=33) Uses a learner-centred approach to teaching, 22% 44% 22% 13% 0% which involves sharing authority with learners. (N=32) Works to nudge students toward the kind of 34% 38% 22% 6% 0% intellectual responsibility they need to become self-motivated, self-actualised learners. (N=32) Encourages the voices of all learners and is 23% 32% 39% 3% 3% respectful of what the learner can contribute to the learning enterprise. (N=31) Assessment should avoid overly narrow 19% 55% 19% 6% 0% measurement of discrete skills or knowledge without a more complete context of interpretation. (N=31) Open-ended discussions based on critical 9% 33% 42% 12% 3% inquiry/authentic intellectual work. (N=33) Builds skills and abilities while working on 12% 33% 42% 12% 0% authentic tasks and problems. (N=33) Provides and balances self-directed and 13% 22% 53%> 13% 0% collaborative learning opportunities. (N=32) Seeks engagement with a community of learners 22% 22% 41% 9% 6% or practice. (N=32) Becomes a metacognitive coach for learners. 13% 41% 25% 19% 3% (N=32) The instructor is flexible enough to support 19% 42% 29% 6% 3% student expressions of different levels of knowledge. (N=31) Should create an environment with a social 26% 26% 26% 23% 0% context. (N=31) Assessment should often include learners' 16% 45% 26% 13% 0% reflections on their goals and accomplishments. (N=31) Builds on prior learning (knowledge) of 33% 30% 27% 6% 3% individuals. (N=33) Critically merges the learner's knowledge of the 18% 36% 36% 6%> 3% world with disciplinary insight and experience. (N=33) Emphasises experiential, authentic learning, 24% 24% 27% 15% 9% which is decided on by learners or negotiated with them. (N=33) Encourages exploration of multiple paths and 22% 19% 38% 22% 0% forms and facilitates opportunities for varying experiences and pathways through the subject matter. (N=32) 131 Social negotiation of meaning can happen 19% 22% 34% 16% 9% through asynchronous / synchronous collaborative tools. (N=32) Assessments should accommodate the needs of 16% 35% 35% 13% 0% diverse learners. (N=31) Learners may take away more from the 26% 16% 26% 26% 6% experience than originally anticipated. (N=31) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Most idea statements in this category were ranked from important to essential by 80% or more of the participant-experts. However, six of the twenty idea statements failed to rank at the level of 80% consensus established for the study. These include the role of the instructor as metacognitive coach (78%); creating an environment with a social context (77%); emphasizing experiential/authentic learning decided on or negotiated with learners (76%); allowing multiple pathways through subject matter (78%); social negotiation of meaning through collaborative tools (76%); learners gaining more than originally intended from the learning experience (68%). Question 3 Scale. The idea statements in the 80-89% consensus category on Question 1 were ranked on the personal/global view scale also. In all, only 11 of the 20 idea statements in this consensus category were ranked higher than 50% on the global scale (see Table 15.3). 132 Table 15.3 Results from Round 3 online vote - Question 3 (80-89%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Learners should have opportunities to solve complex, realistic problems similar to 39% 61% those that the learners themselves would encounter outside the classroom. (N=33) Uses a learner-centred approach to teaching, which involves sharing authority with 41% 59% learners. (N=32) Works to nudge students toward the kind of intellectual responsibility they need to 47% 53% become self-motivated, self-actualised learners. (N=32) Encourages the voices of all learners and is respectful of what the learner can 55% 45% contribute to the learning enterprise. (N=31) Assessment should avoid overly narrow measurement of discrete skills or 48% 52% knowledge without a more complete context of interpretation. (N=31) Open-ended discussions based on critical inquiry/authentic intellectual work. 48% 52% (N=33) Builds skills and abilities while working on authentic tasks and problems. (N=33) 36% 64% Provides and balances self-directed and collaborative learning opportunities. 56% 44% (N=32) Seeks engagement with a community of learners or practice. (N=32) 38% 62% Becomes a metacognitive coach for learners. (N=32) 56% 44% The instructor is flexible enough to support student expressions of different levels 58% 42% of knowledge. (N=31) Should create an environment with a social context. (N=31) 45%> 55% Assessment should often include learners' reflections on their goals and 68% 32% accomplishments. (N=31) Builds on prior learning (knowledge) of individuals. (N=33) 33% 67% Critically merges the learner's knowledge of the world with disciplinary insight and 36% 64% experience. (N=33) Emphasises experiential, authentic learning, which is decided on by learners or 52% 48% negotiated with them. (N=33) Encourages exploration of multiple paths and forms and facilitates opportunities for 53% 47% varying experiences and pathways through the subject matter. (N=32) Social negotiation of meaning can happen through asynchronous / synchronous 44% 56% collaborative tools. (N=32) Assessments should accommodate the needs of diverse learners. (N=31) 58% 42% Learners may take away more from the experience than originally anticipated. 61% 39% (N=31) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%). 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Items reflecting the instructors' roles were ranked low globally, including idea statements reflecting instructor as metacognitive coach, instructor flexibility in supporting varying levels of learner knowledge, instructor balance of self-directed and 133 collaborative learning, learner assessment and its collaborative nature. Items with low global ranking reflecting learners' roles included authentic learning experiences negotiated by learners, learners gaining more than originally anticipated, and providing a voice for all learners and respect for their contributions to the learning experience. Many of these items are consistent with the lower ranked items on the Question 2 scale - in fact there is more consistency with that scale than with the original Question 1 scale. The relatively low ranking on the global scale of many of these idea statements do not indicate their lack of importance as defining elements of constructivism, as determined by the participant-experts. Rather, it indicates that they do not see the individual statements as globally accepted views of constructivism. The literature, however, does support the defining elements delineated in these low-ranked idea statements (Aliponga, 2003; Bopry, 1999; Airasian & Walsh, 1997; Clements, 1997). Consensus 70-79% Question 1 Scale. There are 12 idea statements in this consensus category, none of which attained the consensus level of 80%, which was set by participant-experts for this study (see Table 13.4). Several idea statements in this category suggest the need for flexibility in constructivist learning environments - in particular flexibility in terms of learner assessment protocols, and again, accommodation of shifts in expected learning outcomes. What is noteworthy is that authenticity was also evident in some of the idea statements in this data set. This suggests the idea that learning activities be authentic as well as 134 reflective and that they be housed in a safe environment, in order to encourage risk-taking and dialogue among learners. Table 13.4 Results from Round 3 online vote - Question 1 (70-79%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree Support the goals of learners by facilitating appropriate forms of learner collaboration 78% 22% and discussion. (N=32) Flexible in responding to different starting points for different individual learners. 78% 22% (N=32) Learning activities are "authentic" and require reflection. (N=33) 76% 24% Knowing is mediated by community/culture and the tools of that community and 75% 25% culture, so be aware of and exploit cultural differences. (N=32) Provide activities at the right level of challenge for learners. (N=31) 74% 26% Should accommodate a shift in expected learning outcomes. (N=31) 74% 26% Provides opportunities for dialogue between novices and experts. (N=31) 74% 26% Assessment systems should examine a learner's full range of knowledge, skill, and 74% 26% attitudes toward a domain of expertise. (N=31) Learners should have access to assessment criteria and methods, and be able to negotiate 74% 26% at least some of the terms of the assessment. (N=31) Knowledge does not exist independent of the learner. (N=33) 73% 27% Learners interact with a media-rich and resource-rich environment in building 70% 30% knowledge. (N=33) Provides a safe environment, encouraging risk-taking and authentic dialogue. (N=33) 70% 30% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. There was relatively high agreement on learner collaboration (78%), and on knowledge being mediated by community and culture (75%). However, it is interesting that these two idea statement rankings are lower than one would expect from the literature on social constructivism (Dougiamas, 1998; Cobern, 1993; Wertsch, 1991). Perhaps most unexpected, in terms of just 70% agreement, was the response to the idea statement "learners interact with a media-rich and resource-rich environment in 135 building knowledge." The literature on creating constructivist learning environments presents in great detail the need for rich environments - both authentic and academic (Murphy, 1997b; Honebein et al., 1996; Jonassen, 1991a; Wilson & Cole, 1991). As well "rich media and fresh approaches to learning" are promoted in the literature (University of Western Australia, 2001, 53). Question 2 Scale. The idea statements in the 70-79% consensus category on Question 1 were ranked on the five-point essential - not important scale in Question 2. On this scale, idea statements were for the most part ranked higher than on the Question 1 scale: seven of the twelve items were ranked important to essential in terms of the study's 80% consensus level (see Table 14.4). Five items in this set of data ranked 85-91% as important to essential elements of constructivism. They include flexibility in terms of accommodating individual learners' entry levels, provision of authentic learning activities, provision of activities that challenge learners, knowledge existing only in relation to the learner, and provision of a safe environment to encourage risk-taking and authentic dialogue. These items are all upheld in the literature on constructivism. One item proved to be consistent with the Question 1 ranking in receiving a lower than expected vote. The item "learners interact with a media-rich and resource-rich environment in building knowledge" was deemed to be important to essential by 59% of the participant-experts (it achieved only 70% agreement on the agree/disagree scale). 136 Table 14.4 Result from Round 3 online vote - Question 2 (70-79%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Support the goals of learners by facilitating 6% 25% 38% 28% 3% appropriate forms of learner collaboration and discussion. (N=32) Flexible in responding to different starting points 19% 41% 31% 6% 3% for different individual learners. (N=32) Learning activities are "authentic" and require 18% 24% 45% 9% 3% reflection. (N=33) Knowing is mediated by community/culture and 13% 16% 53% 6% 13% the tools of that community and culture, so be aware of and exploit cultural differences. (N=32) Provide activities at the right level of challenge 16% 39% 32% 6% 6% for learners. (N=31) Should accommodate a shift in expected learning 13% 29% 32% 16% 10% outcomes. (N=31) Provides opportunities for dialogue between 19% 23% 26% 29% 3% novices and experts. (N=31) Assessment systems should examine a learner's 23% 29% 26% 13% 10% full range of knowledge, skill, and attitudes toward a domain of expertise. (N=31) Learners should have access to assessment 26% 32%> 23% 16% 3% criteria and methods, and be able to negotiate at least some of the terms of the assessment. (N=31) Knowledge does not exist independent of the 42% 24% 21% 9% 3% learner. (N=33) Learners interact with a media-rich and resource- 9% 18% 33% 36% 3% rich environment in building knowledge. (N=33) Provides a safe environment, encouraging risk- 24% 30% 30% 9% 6% taking and authentic dialogue. (N=33) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Question 3 Scale. The idea statements in the 70-79% consensus category on Question 1 were ranked on the personal/global view scale also. In all, only two of the twenty idea statements in this consensus category were ranked higher than 50% on the global scale (see Table 15.4). 137 Table 15.4 Result from Round 3 online vote - Question 3 (70-79%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Support the goals of learners by facilitating appropriate forms of learner 69 % 31 % collaboration and discussion. (N=32) Flexible in responding to different starting points for different individual learners. 56 % 44 % (N=32) Learning activities are "authentic" and require reflection. (N=33) 55 % 45 % Knowing is mediated by community/culture and the tools of that community and 69 % 31 % culture, so be aware of and exploit cultural differences. (N=32) Provide activities at the right level of challenge for learners. (N =31) 42 % 58 % Should accommodate a shift in expected learning outcomes. (N=31) 65 % 35 % Provides opportunities for dialogue between novices and experts. (N=31) 58 % 42 % Assessment systems should examine a learner's full range of knowledge, skill, and 68 % 32 % attitudes toward a domain of expertise. (N=31) Learners should have access to assessment criteria and methods, and be able to 61 % 39 % negotiate at least some of the terms of the assessment. (N=31) Knowledge does not exist independent of the learner. (N=33) 24 % 76 % Learners interact with a media-rich and resource-rich environment in building 52 % 48 % knowledge. (N =33) Provides a safe environment, encouraging risk-taking and authentic dialogue. 52 % 48 % (N=33) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. None of the idea statements in this data set attained the 80% consensus level of the study on the global view. In fact, just two idea statements were perceived to represent a global acceptance as defining elements of constructivism: these included provision of activities at appropriate level for learners, and the non-existence of knowledge independent of the learner. While these two idea statements are supported by the literature on constructivism, so are many other items in this category (Winne, 2001; Pintrich, 2000; Brooks & Brooks, 1999). 138 Consensus 60-69% Question 1 Scale. There are four idea statements in this consensus category, neither of which attained the consensus level of 80%, which was set by participant-experts for this study (see Table 13.5). Table 13.5 Results from Round 3 online vote - Question 1 (60-69%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement 1 Agree '""Disagreed Emphasizes co-operative learning (collaborative construction of common projects or 69% 31% representations). (N=32) Constructivism assumes a subjectivist view of the world, even a post-modern one of 66% 34% multiple realities and no "grand narratives." (N=32) Provide a variety of ways of introducing course material so those students with different 65% 35% learning styles are accommodated. (N=31) Provides a clear description of the rules of the course so that all learners understand the 61% 39% playing field. (N=31) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Just two thirds of participant-experts agreed with the idea statement reflecting that constructivism is linked to multiple realities and the subjectivist paradigm. This proved to be a subtle theme that emerged again during the final discussion on the vote in Round 4. Wilson (1997) notes that "In truth, not all construetivists are postmodern in their orientation. In psychology, constructivism originally reflected the thinking of people like Piaget and Vygotsky, who were basically modern in orientation... It is possible to have a constructivist view of cognition while still retaining a fairly traditional, modern view of 139 science, methodology, and technology" (p. 8). This is noteworthy as it continues the ongoing debate within constructivism, of its multiple natures. Question 2 Scale. The idea statements in the 60-69% consensus category on Question 1 were ranked on the five-point essential - not important scale in Question 2. On this scale, idea statements were for the most part ranked similarly to the Question 1 ranking (see Table 14.5). Table 14.5 Result from Round 3 online vote - Question 2 (60-69%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Emphasizes co-operative learning (collaborative 3% 34% 25%> •34% 3% construction of common projects or representations). (N=32) Constructivism assumes a subjectivist view of the 13% 22% 28% 22% 16% world, even a post-modern one of multiple realities and no "grand narratives." (N=32) Provide a variety of ways of introducing course 13% 16% 42% 19% 10% material so those students with different learning styles are accommodated. (N=31) Provides a clear description of the rules of the 10% 45% 23% 10% 13% course so that all learners understand the playing field. (N=31) Note: . 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. As seen in the table, only one idea statement differed to any extent in ranking from Question 1. The idea statement reflecting provision of clear description of the rules of the course, etc., was ranked essential to important by 77% of the participant-experts, while just 61% agreed on the first scale that it was a defining element of constructivism. 140 Perhaps the wording of the idea statement, with its emphasis on clearly delineated and predetermined procedures, was preventive in terms of its inclusion as a defining element. The literature on constructivism is notably lacking in reference to rules and procedural elements of learning environments (Winne, 2001; Driscoll, 2000; Wilson, 1995). Question 3 Scale. The idea statements in the 60-69% consensus category on Question 1 were ranked on the personal/global view scale also. In all, none of the four idea statements in this consensus category were ranked higher than 50% on the global scale (see Table 15.5). Table 15.5 Result from Round 3 online vote - Question 3 (60-69%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Emphasizes co-operative learning (collaborative construction of common 56% 44% projects or representations). (N=32) Constructivism assumes a subjectivist view of the world, even a post-modern one 56% 44% of multiple realities and no "grand narratives." (N=32) Provide a variety of ways of introducing course material so those students with 61% 39% different learning styles are accommodated. (N=31) Provides a clear description of the rules of the course so that all learners 52% 48% understand the playing field. (N=31) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. One idea statement was ranked particularly low on the global scale in this category. The item reflecting accommodation of various learning styles through variation in introduction of course materials was deemed to be a global view of constructivism by just 39% of participant-experts. This low ranking might have its origins in the debate and 141 criticism of learning styles research in recent literature (Sadler-Smith, 2001; Stahl, 1999; Reynolds, 1997; Curry, 1990). Consensus Less Than 60% Question 1 Scale. There are two idea statements in this consensus category, none of which attained the consensus level of 80%, which was set by participant-experts for this study (see Table 13.6). Table 13.6 Results from Round 3 online vote - Question 1 (Less than 60%) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree Steps aside and allow the class to figure out how it will go down the path that is 44% 56% established. (N=32) Practised in the art of evidence-based teaching and instructional design. (N=32) 38% 62% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. A very interesting point in this category is the idea statement suggesting that instructors "step aside and allow the class to figure out how it will go down the path that is established." This idea of mentor or coach, as opposed to didactic presenter of information, is not a new one for most instructors, and it is certainly a central theme in the learner-centred world of constructivism. So why the low consensus level (44%) on this idea statement? It is possible that the problem in accepting this idea statement as a defining element of constructivism lies in its wording. The "step aside" concept may be 142 more difficult to agree on than that of being a facilitator or guide. A l l encourage learners to take responsibility, and do not eliminate instructor creation of a learning environment, but stepping aside may imply relinquishing control of all aspects of the learning environment - something that would be problematic for even many self-professed constructivist educators. Question 2 Scale. The idea statements in the less than 60% consensus category on Question 1 were ranked on the five-point essential - not important scale in Question 2. On this scale, idea statements were for the most part ranked similarly to the Question 1 ranking (see Table 14.6). Table 14.6 Result from Round 3 online vote - Question 2 (Less than 60%) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Steps aside and allows the class to figure out how 6% 22% 25% 34% 13% it will go down the path that is established. (N=32) Practised in the art of evidence-based teaching 16% 3% 31% 28% 22% and instructional design. (N=32) Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Ranking on the five-point scale indicated that 53% and 50% of participant-experts ranked these two idea statements in the important to essential range - just a little higher than the rankings on the two-point agree/disagree scale. 143 Question 3 Scale. The idea statements in the less than 60% consensus category on Question 1 were ranked on the personal/global view scale also. Both of the two idea statements in this consensus category were ranked considerably lower than 50% on the global scale (see Table 15.6). Table 15.6 Result from Round 3 online vote - Question 3 (Less than 60%) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Steps aside and allow the class to figure out how it will go down the path that is established. (N=32) Practised in the art of evidence-based teaching and instructional design. (N=32) 72% 28% 72% 28% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. These two idea statements were disparate in their ranking on the global view when compared with the other two scales, in that their ranking was just 28% each. However, there is a consistency on all three scales in terms of the importance of these two idea statements in defining or representing constructivist teaching and learning. Consensus: Constructivist Learning Activities Question 1 Scale. There are twenty idea statements in this consensus category, representing learning activities that flow from a constructivist belief system, only one of which attained the 80% consensus level set in this study (see Table 13.7). 144 Table 13.7 Results from Round 3 online vote - Question 1 (Learning Activities) Does the idea statement represent an essential core or primary feature of constructivism and online learning? Statement Agree Disagree Content/topic development/resource gathering (N=31) 81% 19% Creative works (N=31) 77% 23% Problem-based learning (N=31) 77% 23% Case-based learning (N=31) 77% 23% Asynchronous discussion (N=31) 74% 26% Portfolios/e-portfolios or collections of work samples (N=31) 71% 29% Guided-discovery learning (N=31) 68% 32% Evidence-based practice (N=31) 68% 32% Webquests(N=31) 68% 32% Independent reading framed as self-guided research (N=31) 68% 32% Live performances (N=31) 65% 35% Inquiry reports (N=31) 65% 35% Performance assessments and skill demonstrations (N=31) 65% 35% Small group inquiry exercises, reported out to the whole class (N=31) 65% 35% Research papers and presentation of findings (N=31) 65% 35% Peer teach ing (N=31) 65% 35% Simulations (N=31) 65% 35% Role-playing (gaming) (N=31) 61% 39% Working models (N=31) 58% 42% Synchronous on-line sessions (N=31) 52% 48% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. It should be noted that an additional three learning activities: creative works, problem-based learning and case-based learning, did attain a fairly high vote (77% consensus). A majority of the learning activities ranked in the 60-69% range, with only two activities, working models and synchronous online sessions, ranking below that level of agreement. Among those activities failing to attain consensus there were a few surprises -namely Webquests (68%), guided discovery learning (68%), inquiry reports (65%) and simulations (65%). These activities are frequently cited in the literature as representative 145 of activities that promote constructivist learning environments (Scardamalia & Bereiter, 2002; Schwier, 2001). Question 2 Scale. The idea statements in the consensus on learning activities category on Question 1 were ranked on the five-point essential - not important scale in Question 2 (see Table 14.7). Table 14.7 Result from Round 3 online vote - Question 2 (Learning Activities) Rank each idea statement on its importance as a primary feature of constructivism and online learning Statement Rank (1-essential - 5-not important) 1 2 3 4 5 Content/topic development/resource gathering 13% 32% 32% 19% 3% (N=31) Creative works (N=31) 13% 26% 23% 29% 10% Problem-based learning (N=31) 16% 39% 23% 23% 0% Case-based learning (N=31) 10% 32% 32% 23% 3% Asynchronous discussion (N=31) 23% 19% 39% 13% 6% Portfolios/e-portfolios or collections of work 13% 32% 26% 26% 3% samples (N=31) Guided-discovery learning (N=31) 23% 23% 26% 26% 6% Evidence-based practice (N=31) 10% 23% 29% 26% 13% Webquests (N=31) 3% 13% 35% 35% 13% Independent reading framed as self-guided 13% 26% 35% 23% 3% research (N=31) Live performances (N=31) 3% 13% 32% 29% 23% Inquiry reports (N=31) 10% 23% 16% 29% 23% Performance assessments and skill 16% 13% 32% 29% 10% demonstrations (N=31) Small group inquiry exercises, reported out to the 13% 29% 32% 19% 6% whole class (N=31) Research papers and presentation of findings 13% 23% 29% 32% 3% (N=31) Peer teaching (N=31) 6% 26% 29% 32% 6% Simulations (N=31) 6% 23% 23% 35% 13% Role-playing (gaming) (N=31) 6% 23% 26% 29% 16% Working models (N=31) 6% 16% 29% 39% 10% Synchronous on-line sessions (N=31) 6% 3% 26% 39% 26% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. 146 On this scale, idea statements were for the most part ranked similarly to the Question 1 ranking. One idea statement did rank at the consensus level set for this study on the Question 2 scale: 81% of participant-experts ranked asynchronous discussion as important to essential. Again, Webquests as representative of constructivist learning activities was ranked low at just 51%, mirroring the ranking on the Question 1 scale. Synchronous online discussions were ranked at just 35% on the important to essential scale. Question 3 Scale. The idea statements in the consensus on learning activities category on Question 1 were ranked on the personal/global view scale also. Of the 20 idea statements reflecting constructivist learning activities, none achieved the 80% consensus level for this study on the global view (see Table 15.7). In fact, only two idea statements achieved 50-59% consensus on the global scale: content/topic development/resource gathering and asynchronous discussions. A l l other learning activities were identified by participant-experts as not universally accepted as constructivist learning experiences, despite the literature having identified many of these activities as part of constructivist classrooms (Pithers & Soden, 2000; Willis & Wright, 2000; Bonk & Cunningham, 1998; Garrison, 1997; Barell, 1991). 147 Table 15.7 Result from Round 3 online vote - Question 3 (Learning Activities) Views represent personal view (minority view) or a global view (majority view) Statement View Personal Global Content/topic development/resource gathering (N=31) 42% 58% Creative works (N=31) 61% 39% Problem-based learning (N=31) 52% 48% Case-based learning (N=31) 52% 48% Asynchronous discussion (N=31) 45% 55% Portfolios/e-portfolios or collections of work samples (N=31) 52% 48% Guided-discovery learning (N=31) 58% 42% Evidence-based practice (N=31) 61% 39% Webquests (N=31) 61% 39% Independent reading framed as self-guided research (N=31) 65% 35% Live performances (N=31) 68% 32% Inquiry reports (N=31) 58% 42% Performance assessments and skill demonstrations (N=3 1) 52% 48% Small group inquiry exercises, reported out to the whole class (N=31) 55% 45% Research papers and presentation of findings (N=31) 61% 39% Peer teaching (N=31) 71% 29% Simulations (N=31) 71% 29% Role-playing (gaming) (N=31) 77% 23% Working models (N=31) 65% 35% Synchronous on-line sessions (N=31) 68% 32% Note: 1. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 2. Because of rounding of the initial numbers, some totals are 99% or 101%. 3. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. Summary Overall, this round created some intriguing data for examination. Two of the idea statements put forward by the participant-experts did achieve 100% consensus, while other idea statements did fall within the 80% + consensus boundaries applied to the process by the participant-experts. As noted earlier, the two idea statements that did receive 100% consensus reflect current thinking in the area of learning theory and are supported by the current literature. The fact that these two idea statements achieved 100% consensus as representative of defining elements of constructivism, across all participant-148 experts in the study, indicates their acceptance as core to constructivist learning and teaching. Beyond the 100% consensus, much of the remaining data was also consistent with the literature (i.e. the high level of agreement on the social elements of constructivist learning, of human interaction, and of valuing multiple perspectives). There were also some unusual anomalies that diverged from much of what the literature has been reporting. For example, it was interesting that 'learner-centeredness' did not achieve the complete consensus on the global scale, as might have been expected, while many of the activities promoted in the literature as being constructivist in nature were not noted by higher consensus. In reviewing the idea statements that achieved consensus, they can be placed under the eight major themes that emerged from the original discussions. Under the category /theme of Instructional Design, five idea statements emerged: the core of the design is learner-centred (100%); open-ended discussions based on critical inquiry/authentic intellectual work (85%); should create an environment with a social context (84%); provides and balances self-directed and collaborative learning opportunities (84%); and encourages exploration of multiple paths and forms and facilitates opportunities for varying experiences and pathways through the subject matter (81%). Under the theme of Social/Community/Culture, two idea statements emerged: learners are encouraged to seek knowledge and experience from different types of sources and perspectives (91%); and social negotiation of meaning can happen through asynchronous / synchronous collaborative tools (81%). 149 Under the theme of Environment, nine idea statements emerged: supports multiple conceptions and promote critical thinking that stretches boundaries (97%); fosters active/higher-level thinking and reflection (94%); emphasize/facilitate active participation of learners with the content (dialogue) and with each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group (94%); offers opportunities for students to follow their own interests (91%); learners should have opportunities to solve complex, realistic problems similar to those that the learners themselves would encounter outside the classroom (88%); encourages the voices of all learners and is respectful of what the learner can contribute to the learning enterprise (87%); builds on prior learning (knowledge) of individuals (82%); emphasizes experiential, authentic learning, which is decided on by learners or negotiated with them; and critically merges the learner's knowledge of the world with disciplinary insight and experience (82%). Under the theme of Instructors, five idea statements emerged: provides opportunities, but not always answers (91%); works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, self-actualized learners (87%); uses a learner-centred approach to teaching, which involves sharing authority with learners (87%); the instructor is flexible enough to support student expressions of different levels of knowledge (84%); becomes a metacognitive coach for learners (84%). Under the theme of Learners, four idea statements emerged: sharing multiple perspectives and hearing the constructions of others allows one to question our original assumptions and to develop a shared meaning (91%); builds skills and abilities while working on authentic tasks and problems (85%); seeks engagement with a community of 150 learners or practice (84%); and learners may take away more from the experience than originally anticipated (81%). Under the theme Activities, one idea statement emerged: content/topic development/resource gathering (81%). Under the theme Assessment, three idea statements emerged: assessment should avoid overly narrow measurement of discrete skills or knowledge without a more complete context of interpretation (87%); assessment should often include learners' reflections on their goals and accomplishments (84%); and assessments should accommodate the needs of diverse learners (81%). Under the theme Theory, four idea statements emerged: encourages/values the development of personal understanding/knowledge construction by the learner (100%); the process is iterative and lifelong so that our own "meaning-making" is constantly changing (97%); understanding that there are multiple ways that 'knowledge' is absorbed and constructed (97%); and learning is constructed as a result of the process(es) where multiple perspectives are revealed, discussed, then reconstructed (91%). In all there were 51 idea statements that were reviewed by the participant-experts and 33 achieved a consensus of over 80% as defined by the participant-experts during the study. This left the remaining idea statements below the 80% consensus level, including all but one of the discussed activities that might lead to a constructivist environment. Thirty-three participant-experts out of 38 participated in this part of the study. 151 CHAPTER 6 DISCUSSION This chapter has two thrusts: a discussion of the research questions and of the NGT as an online research approach from the perspective of the participant-experts. It ties the results of the NGT process, and the final consensus of the participant-experts to the original research questions and the literature in the field. It highlights the findings of my research, in particular those findings that either challenge or build on the current literature. In conducting this study I asked the following questions: what are regarded as the primary features, characteristics, elements of constructivism as a theory of learning?; what approaches or procedures to instruction in relation to online learning seem to flow from constructivism as it is currently conceived?; and given these characteristics and procedures, what, do you believe is the role of the instructor? These questions guide the discussion. Round 4 Data In this Round, the voting results from Round 3 were presented for viewing and comment by participant-experts. The participant-experts were required to comment on the results of the vote, to raise points about any surprise or unexpected outcomes, and to offer explanations for their agreement or disagreement with the results. They were also asked to indicate their views on the issue of consensus - that is, what was the percentage of agreement they required as an indication of consensus on the item statements. Finally, they were asked to affirm their agreement - or consensus — with the voting results. 152 The discussion was available to all 38 participants. It was divided into four threads, with each thread initiated through an opening posting from the facilitator covering one of the questions proffered in this Round. In all, the participation rate for Round 4 was 55%, with a total of 75 postings throughout the Round. Comments on Round 3 Vote There were some interesting comments made by participant-experts regarding the vote on many of the idea statements. The discussion ranged from the general issue of classifying constructivism and its meaning for teaching and learning, to specific classroom implications for those operating within a constructivist framework. The overriding stance to emerge from the Round 4 discussion was the ambivalence in classifying constructivism, in terms of viewing constructivism as a philosophy as opposed to as a learning theory. This debate is reflected in the literature on the subject (Wilson, Osman, Jouchoux, & Teslow, 1997; Wilson, 1997; Young, 2003; Airasian & Walsh, 1997; Fosnot, 1996; Doolittle, 1999). Many participant-experts believe that constructivism is a philosophy or epistemology of learning, as opposed to a learning theory. This discussion began with the research question, what are regarded as the primary features, characteristics, elements of constructivism as a theory of learning? The argument for a philosophical stance is represented in the following postings: Participant 33:1 am not sure that constructivism is a theory of learning. It is certainly an epistemological viewpoint, and may be seen as a pedagogical philosophy. The concept of constructivism is too broad to be seen as a theory of learning, 153 Participant 5: Constructivism is a philosophical perspective that has gained a favourable rating by many in the education field... I believe that constructivism represents a blend of a number of philosophical and educational theories...Perhaps it is a very complex theory or philosophical perspective that makes it difficult to study. Participant 10: To my mind, constructivism is the philosophical bedrock upon which learning theories are built. Participant 25: At a fundamental level, I think that constructivism is a philosophical stance one adopts as an instructor - not a set of activities, actions or pedagogical approaches. While I considered most everything we've discussed in this study as consistent with a constructivist philosophy, I hesitated to say that most of them were absolutely critical components of a constructivist philosophy. Consistent with the ambivalence about constructivism being narrowly defined or classified as a learning theory, participant-experts expressed understanding that constructivism, as a belief framework, does not require narrow adherence to certain classroom practices (Ally, 2004; Jonnassen, 1991a; Murphy, 1997b). The point was made that even didactic approaches, at times, may be tolerated, and if required by the context, encouraged, within the boundaries of constructivist beliefs. As one participant-expert (Participant 1) indicated, constructivist beliefs often are not held by other colleagues, and because of large classes and basic level courses, approaches deemed to reflect constructivism are not always practical. I have found this within my own practice as an instructional designer. I have found it is also possible to move beyond this barrier of practicality within larger or beginner classes. Participant 35: One of the concerns that I have about the overall results is that it seems the term constructivism has become a catch-all for every approach to teaching and learning that is currently trendy - problem-based learning, collaborative learning, critical thinking, etc. etc. These are all methods, and approaches that have been used for years, and do not necessarily require a constructivist philosophy. For me, the core of constructivism is an approach that allows learners to construct meaning. This can be done with a variety of teaching 154 approaches, some can be very teacher - directed even Socratic, others very learner-centered. Participant 21: Thank you, #35. I too tire of constructivism meaning any set of methods other than didactic presentation. I wholeheartedly agree that instructor leading or giving direction is not ruled out in constructivism - in fact some direction is probably a requirement. Participant 25: What I am trying to say is that I think you can still be a constructivist without doing each particular thing [learning activity] - or perhaps any of them. It seems very likely that a constructivist instructor would do many or most of these things in order to be consistent, but are they absolutely necessary? Participant 4: About activities, what I get from this data is that i f you want to be constructivist you can use basically anything, except maybe overlong frontal lecturing.... Does it surprise anyone else that "active participation of learners" was felt to be essential by only 45% [of participant-experts]? Related to the above discussion is the notion of learner-centeredness. Is learner-centeredness an essential defining idea of constructivism? The majority agreed, but also felt that it is possible to be learner-centered in the classroom without being committed to a constructivist philosophy of teaching and learning. I believe that this means that the concept of learner as the center has found its way into many classrooms, within the North American context, without a full 'buy-in' to constructivism as foundational mode of interaction. Participant 6:1 agree that it is easy to slip from the idea of construction (emphasis on philosophy or framework) to learner-centered instruction (emphasis on strategies). Participant 13: Looking at the results, I'd certainly agree that many of us (me included) slipped into answering the questions according to what we thought was good educational practice, rather than considering a narrow or dogmatic definition of constructivism for each one. It appears that "constructivism" has become a placeholder in our minds for any learning experience deemed progressive or learner-centred. 155 Participant 20: The instructor who takes a learner-centered approach to teaching can use a variety of instructional strategies to achieve the goal ... which is to facilitate learners constructing meaning from their experiences. Participant 26: ... A number of the items below the 78% agreement level represent a focus on "learner-centred instruction" rather than on constructivism per se. These overlap, but some learner-centred items seem to be more from the adult education literature... How much learner control and freedom to explore, in terms of providing an acceptable balance in a constructivist environment, was the subject of discussion. This discussion encompassed the research questions of what approaches or procedures to instruction in relation to online learning seem to flow from constructivism as it is currently conceived?; and given these characteristics and procedures, what, do you believe is the role of the instructor? Learner control, as reflected in the idea statements, ranged from control in exploring content to control in setting their own objectives and determining suitable assessment activities. This discussion also finds alliance with the current literature (Bull, Montgomery, & Kimball, 1999; Jonassen, n.d.; Seels. 1989). Finding a balance is not easy, but most participant-experts agreed that learner control of learning experiences is essential to being constructivist - with limitations. Participant 24: But when we want our students to engage in more critical thinking and deep understanding about a subject, we need to use more learner-centred approaches like problem-based learning etc. Leading or giving direction can be done in such a way that it is not didactic but more like metacognitive coaching. Participant 5:1 have a concern.. .involving the degree of control and guidance that the online teacher should have in the area of assignments. It is interesting to note the discovery method is very time consuming for students. One of the [idea statements] in Round 3 spoke of teachers stepping aside, but there is a point that students need guidance, even doctoral students! I would argue against excessive use of constructivism in the online class. I have taught numerous graduate students and they need clear guidance and examples to assist them. 156 Participant 35:1 agree strongly that constructivism does not rule out the instructor providing guidance.... Participant 15:1 don't know whether we were supposed to have our worlds transformed or shifted - but I do think that the activities that have fallen out of a constructivist perspective on teaching have transformed the world of teaching and learning over the past 20 years. Constructivist learning environments are not the same as the places where I learned - and that's a good thing, in my humble opinion. Consensus on Consensus Level In determining consensus level, there were few guidelines in the literature that were useful to my study. Through my reading and in consideration of the size of my panel, I as facilitator had initially considered that an 80% vote would be adequate to assume consensus. However, in keeping with the premise of the NGT as a process that encourages input and discussion on the part of participant-experts, I wished to establish consensus through consultation. Participant-experts did provide guidance on this matter, and in fact 80% was deemed by the majority to be the minimum vote level for an idea statement to have achieved consensus. Participant 26: It is not uncommon for 80% to be used as a competency cut off level for licensing exams in Nursing, Teaching and so on, so it's a generally accepted number. Participant 23:1 thought I would jump in and say that I agree with the arguments re setting the threshold at 80%. This seems reasonable and captures critical items that might otherwise not be included. Participant 17:1 vote 80%. You will never make everyone happy and i f we can agree, we can move onto the next part. The discussion on establishing consensus was interesting, and revolved around the wisdom of seeking consensus in such a process as the NGT, and the use of absolute percentages as an indicator of consensus. While the majority was in agreement generally 157 on the issues surrounding consensus, some participant-experts opposed the 80% consensus benchmark. One participant (estudy26) proposed setting the consensus level at 82%, but including four idea statements with an 81% vote because, on the global/personal scale, they had a high global vote. As with the majority of the participant-experts, I was satisfied with the setting of consensus at 80%. Participant 6: I'm fine with the 80% cutoff. Moving below 80% disenfranchises larger numbers of minority perspective, so I hesitate to go much lower than 80%, even if we're leaving out some good concepts. Participant 24:1 think the 80% shows some consensus, but is it strong enough for the purposes of this project? I lean towards raising the bar to at least 85%. Participant 12: Consensus is unanimity: as a group we have consensus on two points [idea statements], and only on two. I vigorously disagree with calling an 80% majority opinion "consensus." No matter what the requirements are for this particular research method, I do not agree with calling a result a consensus when it is clearly not a consensus. However, I do agree that points [idea statements] on which achieved at least 80% agreement are points [idea statements] on which the group as a whole had a strong convergence of opinion. Participant 20:1 think using an 80% agreement level allows us to include those items which have, a high priority in terms of constructivism. However, with all but two of the item statements, there are from one to six colleagues who disagree. The question I would like to ask if those 1-6 people are okay with agreeing to disagree in this consensus-building technique. If so, I think we've done well. Participant 35: Do we really gain anything but artificially defining "consensus?" The results clearly indicate what most people think constructivism is. Isn't that meaningful enough? Setting a cut-off point is an attempt to quantify something that can't be quantified that precisely. Affirmation of the Consensus Vote Many of the participant/experts did not affirm their own consensus with the overall vote on the idea statements. But of those who did, they were, for the most part, in agreement with the vote on each item. However, they did comment on items for which 158 they were surprised at either the lack of support or more support than anticipated, and in particular items where there was inconsistency in percentage votes across the three scales: the agree/disagree scale, the essential/not important scale, and/or the global/personal view scale. Participant 18: Relating to culture and context, I wasn't surprised ... that 84% of the group responded that constructivist theory promotes creation of a learning environment" with a social context. I was surprised that only just over 50% of the group felt this was a "global" opinion rather than simply their own [opinion]. Participant 33: Then I read the results of the vote, and saw that only 66% agreed with the idea statement: Constructivism assumes a subjectivist view of the world, even a postmodern one of multiple realities and no 'grand narratives.' Does this mean that some people think that constructivism is right, ignoring its postmodern origins? Participant 25: The 'subjectivist' claim for constructivism received a two thirds vote. That surprised me since I don't see constructivism relying on a subjectivist foundation, and I didn't know I was in the minority. Now I know! Participant 15: What does interest me is that idea statements that I thought were the heart and soul of constructivist teaching were not receiving higher global percentages. Does it surprise anyone else that "active participation of learners" was felt to be essential by only 45%, and that 3% thought it to be not important? Participant 21:1 too was surprised by the relatively lower levels for the global view in some cases - i.e. 31 % Knowledge is mediated by community/culture etc: 67% Builds on prior knowledge of individuals etc: 44% Constructivism assumes a subjectivist view etc. Summary In all, the discussion of the Round 3 vote and of setting consensus levels proved to be of interest. Postings were informative, and many postings were followed up by other participant/experts, creating a dialogue. While there were minor disagreements on the vote for some of the idea statements, for the most part there was evidence that the majority were aligned in their thinking on constructivism. 159 Online NGT The online NGT proved to be feasible, overall. The process worked through all of the Rounds, and the majority of participants remained dedicated to the study over the extended period of more than four months. The process resulted in valuable data on the research questions. The online nature of the NGT made it feasible to implement debate and discussion among participants across broad geographic areas, and did not seem to hamper the depth or quality of the discussions to any significant extent, although the sheer magnitude of the idea statements presented did create problems for participants. In all, the online NGT could be deemed a success. There were both problems and successes in implementing the online NGT. As researcher I kept a diary of the implementation process, noting things that worked well and any problems that arose. In addition, the End of Process survey sought participant views on the process itself. The following discussion is based on my observations, and participant observations of the online NGT process. Participants were, for the most part, positive about their online NGT experience. They did point to various specific flaws in the process as implemented, but overall they voiced approval for the process. Of participants responding to a query on NGT as a consensus building activity that creates opportunity for generating and encouraging discussion, the majority (50%) found the experience beneficial rating it very useful/useful (see Table 16). 160 Table 16 Participant Opinion of Online NGT as a Discussion Vehicle Participant Opinion (n=27) Percentage Very Useful/Useful Somewhat Useful Not Useful 50% 26% 22% Participant comments on the effectiveness of the online NGT included: Participant 5:1 would [like] it over a shorter period of time to help me [get] more immersed in the process. I work best when I am able to have more focused attention [on] a project. Participant 6: In the online world the methodology gets in the way of discussion. There are too [many] rules and too many repeated or redundant questions, and the participant (this one) gets annoyed... It made me want to play games with the questions rather than be a serious contributor. Participant 18: The asynchronous nature of the activity made it very difficult for me to participate actively and to get a sense of ongoing conversation. Participant 27: It seemed like a very natural thing to do and I like the negotiation of meaning and interpretation among peers. Participant 35:1 think at least some part of the process needs to happen in real time, preferably face-to-face. A few participants misunderstood the actual intent of the NGT, which has a very structured framework and is a very prescriptive process. It is not meant to encourage free-flowing and open conversation. They noted that conversation "was too directed and structured" and that "thinking was directed too much by the process." However the NGT, whether online or via traditional mail, is designed to impose structure on participants as they communicate. I have learned from this study that in using a structured model such as an NGT, more detailed explanation of the structure is necessary. 161 Participant Commitment Participant commitment was very strong. There was no problem in assembling a panel of committed participants, and in fact nearly half of all those initially approached volunteered to take part in the study. According to participants, they spent an average of more than four hours devoted to the NGT process, and these are all very busy professionals. However it should be noted that the high participation rate over the lengthy process may be attributed in part to my frequent reminders to participate sent via email at least twice in each Round. Without these frequent email messages from me as facilitator, it is doubtful that participants would have maintained their commitment to respond in each successive Round. Group Size of Online NGT The major error in thinking on my part in implementing the online NGT was group size, or number of participants. I initially thought in terms of a group of twenty to twenty-five, but there was an assumption on my part that participant attrition from the study over all Rounds would be significant. Hence I created a panel of nearly double my original number - i.e. 38. My assumption regarding attrition proved to be fallacious, and in fact attrition proved to be insignificant. As a result the large group of participants was unwieldy and affected the quality of the discussion from the beginning, to the extent that it became necessary to sub-divide into smaller groups as the process moved forward. Over Rounds 2 and 2.2,1 reduced the size of the groups by creating first two parallel groups, and then eleven small groups, the latter of which dealt with a small number of the original idea statements. Breaking the panel into small groups worked, in 162 that participants could discuss in some depth their thinking on the issues. In retrospect, it would have been better to have begun with a panel of twenty participants, and i f a significant number of participants dropped out of the study, to have added a second separate panel of twenty participants in a parallel study. Participants were frustrated by the group size, and hence the great quantities of reading that were generated by having so many participants. Until groups were broken down, participation beyond Round 1 was negligible. Participants noted their frustrations with operating in a large panel: Participant 12:1 did learn that, were I to use this technique, I would be very careful to limit the number of participants. Participant 9:1 think that for this technique to work, a small size group is necessary. Discussion is not very good in such a large group as we had. Participant 29: There was some difficulties managing and discussing in the large group...worked better when [put] in smaller groups. This frustration has been seen by me, in past online teaching, where class sizes have been greater than 20-25 participants. It also bears out some of the early literature on optimum class sizes for online participation. Round 1 Brainstorming This Round produced over 400 idea statements on constructivism, which proved to be impossible for panelists to discuss in any meaningful way. Yet one of the strengths of the NGT is the creation of a list of ideas for discussion by the participants themselves. In retrospect, participants should have been asked to provide a maximum of five idea 163 statements, and with a reasonable size panel of twenty, the result would have been a maximum of one hundred initial idea statements. Participants, too, found the number of idea statements generated difficult to deal with. It affected their ability to have any depth of discussion, and in fact it was the most negative element of the process cited by them in the End of Process survey. Representative comments include: Participant 31: The completion of long lists [of idea statements] was not easy and off-putting. Participant 21: The number of ideas presented at the first was overwhelming...too many in the N G , and too many definitions/essential elements. Participant 4: The huge amount of information that sometimes was generated, because it was hard to read everything online and to keep track of the discussion. Time Line of the Online NGT At the beginning of the process, I naively thought that each Round could be completed within one week, and that my analysis of data between Rounds would be completed within a few days. I did not anticipate the overwhelming piles of data that emerged from each Round, nor did I realize the effect of personal commitments that participants might have that interfered with their timely input in discussions. In fact the data analysis between Rounds was enormous, especially at the beginning of the study. Round 1 alone produced over 400 idea statements that had to be understood, merged where necessary, and assigned to themes without losing the original intent of the participants. Increased time for my analysis as facilitator between Rounds 164 resulted in the extension of the N G T time line for all participants, and hence continuity of discussion and ideas was lost. Participants work commitments and schedules were also problematic, in a process that spanned more than four months. In addition, the NGT time line extended through the summer and travel/holidays proved to be disruptive for some participants. Participants (60%) found it difficult to maintain regular engagement in the process over the four months (see Table 17). Table 17 Participant Engagement over the Four-Month NGT Process Difficulty in Maintaining Engagement (n=25) Percentage Difficulty regularly in maintaining engagement 60% Some difficulty at times in maintaining engagement 36% No difficulty in maintaining engagement 4% Participants expressed frustration with the extended time line of the study, as indicated in the following sample statements elicited in the End of Process survey. Participant 3: Longer time lines...spring and summer are hectic times to do this...[I was] busy finalizing courses for fall or away on summer vacation. Participant 8: It did seem to last a long time and it was difficult at times for me to jump into the discussions due to my busy schedule. Participant 19: The length of time was frustrating. Our time frame had to be extended because of the amount of data coming in. . . the same thing happened when I conducted the online Delphi... 165 The Technology - Platform/Tool Choices WebCT was the choice of platform for the online NGT, supplemented by HostedSurvey in Round 3 and the End of Process survey. There were very few problems in accessing or moving around these tools throughout the whole process. Only three participants experienced any difficulties, and their problems were for the most part easily remedied. However WebCT proved to be less than ideal, in terms of capturing the free-flowing, open discussion that was desired by the facilitator and the group. The large number of idea statements initially generated exacerbated this problem, and the subsequent need to try to discuss so many and varied points of view. But in all, WebCT did prove to have some positive attributes for this study, in that it proved to be a familiar platform for most participants, and easy to access and navigate, with 77% reporting their WebCT access went very well/well, (see Table 18) Table 18 Participant Opinions Regarding Access to WebCT Platform Relative Ease of Access (n=26) Percentage WebCT Access Went Very Well/Well 77% WebCT Access Went Somewhat Well 19% WebCT Access Did Not Work Well 4% Participants were also asked to express their opinions on WebCT as a platform for online research involving discussion processes such as NGT. In all, 59% were positive about its applicability to such research studies, (see Table 19) 166 Table 19 Participant Opinions of WebCT as an Online Research Platform WebCT as an Online Research Platform Worked (n=26) Percentage Somewhat Well Not At A l l Very Well/Well 59% 27% 14% Participants expressed dichotomous views of WebCT as applied to the online NGT study, based mostly on this one experience. Hence some of the views may have been coloured by other elements of the study, such as group size and magnitude of data. Participant 5:1 like the way WebCT can be divided into different discussion areas and then divided into specific threads, so discussion is easy to follow. Participant 9: Change the use of WebCT because not all ideas can be seen at once.. .have to spend a lot of time reading through each posting, and when the Internet connection is very slow it becomes very time consuming. Participant 14:1 don't like the discussion tool in WebCT but it is a familiar environment. Participant 18: The WebCT discussion forum tool is not very conducive to developing an online conversation. It was not easy to follow the threads. Participant 29: Easy to use - privacy appears to have been maintained; assume that it collated much of the data quickly. HostedSurvey worked well in Round 3 and in the End of Process survey. However in the latter I was limited by the parameters of the tool. I had intended to use a triple matrix for each survey item - i.e. three response categories (agree/disagree; rank order on five-point scale; global/personal opinion/view). The tool provided for a maximum of two matrices. As a result I had to use the double matrix for each item, and 167 follow the item immediately with a single matrix. That procedure allowed all three matrices to be visible to participants as they accessed each item - an awkward but utilitarian solution. The majority of participants (85%) experienced no difficulty in accessing or using HostedSurvey. (see Table 20) Table 20 Participant Experience in Using Hosted Survey Ease of Accessing HostedSurvey Percentage No Difficulty 85% Some Difficulty 15% End of Process Survey Worked... Percentage Very Well/Well 65% Somewhat Well 31% Not At A l l 4% Participants expressed their approval of HostedSurvey in many positive comments, represented by those presented here. Participant 4: The tool can identify patterns and differences within the group, and help researchers develop theories. Participant 10: It saved results in small chunks and therefore [there was] less chance of losing work. Participant 16:1 think this survey tool works well - at least it hasn't crashed on me. 168 Summary of Online NGT Process This study was exploratory in terms of applying a survey process (the NGT) in an online environment. There was little guidance from the literature to direct my study because applications previous to my study were scant. The process as implemented by me was fraught with difficulties at times - however, many of the difficulties were generic to the N G T itself - whether online or traditional application. Size of panel, number of idea statements initially generated, extension of the original timeline to accommodate the participants and the facilitator's workloads were not features of the online nature of this NGT implementation - these problems would have surfaced in a traditional use of the process as well. In terms of the online NGT as a research tool, the process did work - albeit with problems - and data derived from the process were of value in addressing the research questions of my study. The process had some rewards for participants as well: many expressed their enjoyment of the opportunity provided to discuss constructivism, online learning, and instructional design issues with their peers. It is hoped that my experience in implementing the NGT online will provide some guidance for those who wish to implement such a study in the future. 169 CHAPTER 7 S U M M A R Y , CONCLUSIONS A N D RECOMMENDATIONS The purpose of this study was to identify the principal tenets/presuppositions of constructivism and to suggest processes or instructional approaches that flow from this for the development of online learning. This chapter includes a summary of the study, as well as conclusions and recommendations that resulted from it. Summary of the Study Between April and September 2004,1 conducted an online consensus-building exercise in which 38 participant-experts from 9 countries and 4 continents, discussed, debated and attempted to achieve a consensus on constructivism and online learning. In undertaking this study it was my intention to have participant-experts come to a consensus on 'the principal tenets/presuppositions of constructivism as a theory of learning, and to suggest processes or instructional approaches that flow from this theory for the development of online learning' including the role of the instructor within this environment. It was also my intention to use a methodology that had traditionally been used face-to-face (the NGT) to investigate its potential to be used successfully in a wholly online environment. WebCT and HostedSurvey, an online commercial survey tool, were used to conduct the exercise and I acted as facilitator throughout the process. From an original set of over 400 idea statements, themes and sub-themes were identified and fed back to the participant-experts over four Rounds, which included private postings, small group 170 discussion, an online voting procedure and finally, a group discussion that included all of the participant-experts. Between each Round, data were examined and returned to the participant-experts for the next Round of discussion or vote. This resulted in a final set of 51 idea statements. Of the final 51 idea statements identified in the study, 33 achieved a consensus of 80% or higher. Participant-experts moved us closer to a consensus based agreement on what constitutes constructivism and offered several idea statements on how an instructor might consider assisting in the creation of a constructivist friendly classroom and what instructional approaches might facilitate this environment. Conclusions The data that was gathered from the NGT online process was very rich and abundant. From that data the following conclusions are taken. Conclusion One The views of the NGT participant/experts were consistent for the most part with the current literature on constructivism and online learning. It has been suggested that this would not be an unexpected result given many of the writers of the literature participated in my study. This study was an attempt to discern if, beyond the literature, a group of experts could arrive at a consensus. While doing so, they remained consistent with the current literature. 171 Conclusion Two Participant-experts did not appear to differentiate between online learning and the face to face classroom when discussing constructivism. This may have been possible as most of the participant-experts worked in the area of online learning in some capacity. It became apparent that they did not see the delivery vehicle (the Internet versus the face to face classroom) as being a core issue. The discussions ranged on constructivism and learning. This suggests that for some educators, at least, the line between using technology to deliver instruction (the Internet, e-learning, distance education) is quickly becoming less of a division and more of a continuum. Further research is needed in this area. Conclusion Three There was some tension within the group of participant experts with respect to constructivism being a learning theory or a philosophy of knowing, with it being viewed more broadly than a learning theory. This tension remains in the literature and was not clearly reconciled within this study. However, another issue that arises here is the possibility of the importance of the underlying philosophy or values of the instructor and how it might impact on choices within an instructional setting. This core philosophy appears to influence the instructor and has a definite affect on the learning environment, the learning and the teaching. This core, from the discussions in this study, appears more important then a discussion on constructivism and technology, allowing the delivery vehicle to change while the core remains constant. 172 Conclusion Four In working through the data, the issue of what constituted consensus for the purposes of this study arose. The participant-experts, in Round 4 set it at 80% after some debate. This percentage remains somewhat arbitrary and is open for review. This choice to set the consensus at 80% did leave out many concepts advanced in the literature as being inherently constructivist. It was unusual to ask participant-experts to arrive at a 'consensus on consensus' but it was possible to do this in the online version. I am not sure this would have been as successful in the traditional NGT. Conclusion Five One of my intentions was to identify concrete suggestions about the instructional implications of constructivism in an online environment. In my attempt to achieve consensus among the participant-experts, I tried to get consensus on various activities, which the literature had suggested, were constructivist in nature, but most didn't achieve consensus. It would seem that the participant-experts were more comfortable dealing with constructivism at the mental/ideas level than on the practical/application level. Recommendations from NGT A number of recommendations resulted from the use of the NGT in an online environment. 173 Preliminary Preparation Recommendation 1: The facilitator should provide an initial tutorial in NGT as a process - its procedures and intents, so that all participants understand the process in which they are to be involved. This turned out to be critical in participant-experts willingness to stay with the process, as it became longer than intended. Participant Commitment Recommendation 2: The online NGT facilitator should ensure that participation is monitored on a daily basis, so that action can be taken immediately i f Rounds are moving slowly or participation is low. Recommendation 3: In implementing an online NGT, the facilitator should send frequent messages of encouragement and reminders to participate while each Round is active, and between-Round reminders of the beginning date for the next Round. I found the monitoring and frequent contact with participant-experts to be significant in maintaining participant commitment and the integrity of the study. Group Size of Online NGT Recommendation 4: Limit size of panel in the NGT to a maximum of twenty participants. This recommendation is vital when conducting an online discussion-based methodology such as the NGT. Discussion becomes difficult and incoherent when the participant numbers exceed a workable level. Recommendation 5: Once ideas are generated, for in-depth discussion that follows, consider breaking the main group down to groups of eight to ten participants. 174 Groups can be re-combined toward the end of the round to review each other's work (discussion outcomes). Round 1 Brainstorming Recommendation 6: Despite the desirability to have idea statements on the topic freely generated by the participants themselves, limit the number of original idea statements submitted by participants to a maximum of three to five key statements each, depending on the size of the original panel. Recommendation 7: If the number of ideas presented affects the quality and depth of the discussion, consider breaking the panel into smaller groups, with each group handling a limited number of idea statements. Time Line of the Online NGT Recommendation 8: Anticipate the amount of data and the time needed for next Round preparation so that time lines do not have to be extended. Recommendation 9: Develop a complete and realistic schedule for all parts of the NGT that require active engagement of participants, including the End of Process survey, and provide participants with the schedule when they agree to participate. Recommendation 10: Plan to complete the full NGT process, including the End of Process survey, within a maximum period of two months. 175 The Technology - Platform/Tool Choices Recommendation 11: For survey portions of online NGT studies in the near future consider using HostedSurvey as it works well, costs nothing for initial online survey exploration, provided survey numbers for the whole study are under 250, and is easy to access and use for participants. Recommendation 12: Explore alternatives to WebCT, particularly alternatives that will allow more natural chat/conversation structures and free flow of ideas. Further Study of the Online NGT Recommendation 13: Further studies should be implemented using the NGT online, resulting in greater direction and guidance for researchers in the future. Further Study of Constructivism and Online Learning Recommendation 14: My study showed no differentiation by the participant-experts between instruction in online and face to face learning. This is an area that should be explored to see i f in the minds of most educators the boundaries are collapsing between distance education/e-learning and live education/learning. Recommendation 15: There should be further study on the role of the instructor in constructivist online learning environments. Recommendation 16: There should be further study on the translation of constructivism into applied practices. 176 Summary Rationale for NGT My final use of the Nominal Group Technique (NGT) wholly online was a decision I did not make lightly. Encompassing all of the issues for my decision noted in Chapter 3, was my overriding rationale for using the NGT, that of curiosity. The NGT is an interesting methodology and had been used so little in the online environment that I proposed to place it within. My research curiosity took me to the question of how would the NGT function in an online environment; a question which was somewhat risky as I had very little literature or past practice to guide my choice. This resulted in some very concrete recommendations on how future researchers might utilize an online NGT. One of the critical questions asked about the NGT is 'could not the same results have come from an extensive review of the literature, especially since much of the literature was written by those participant-experts engaged in this study?' I considered this question carefully. My research was to investigate the possibilities of consensus and while the literature might have a reader suggest some consensus, it should be understood that academic literature is often more of a debate on an issue than an attempt at consensus. The authors of the literature are not looking to achieve consensus but to offer up suggestions and ideas that, individually, push the thinking of the reader. My intention was to bring together many of the current writers and thinkers to see i f an actual consensus, outside of and because of their writing, was possible, and the NGT made this a viable process. My conclusion is that the NGT has excellent potential as an online methodology when consensus is the goal of the research. 177 Toward a Theory of Constructivism Over several months, I interacted with and facilitated the involvement of 38 participant-experts through the online NGT with a view to shedding light on a more detailed description of constructivism with respect to designing online learning and the instructor's role. The data did not result in a detailed theory or more robust conceptualization of constructivism although it did offer a number of core ideas that are consistent with current literature. Fundamentally the data was not theoretical enough to result in a newer detailed theory. That will be left for future researchers to focus their research in that direction. What was gathered were ideas that led to a more practical description of what constructivism is, as described by the participant-experts. The following description, emerging from the study, uses the language and concepts employed by the participant-experts in the study: Constructivism encourages and values the personal understanding/knowledge construction by the learner and the design of learning is student-centred. Understanding that there are multiple ways that knowledge is absorbed and constructed, there are a number of conditions that should be encouraged that were felt to be essential to constructivism. Learning is iterative and lifelong; it fosters active, higher-level thinking and reflection; it is social in nature. It builds upon shared negotiated meaning and upon the learner's interests. It is situated within realistic authentic tasks and complex problem solving, based on and found in the world outside of the 'classroom'. Instructors share the authority and assist 178 students in adopting intellectual responsibility for their own learning. Instructors are flexible in their support of student expressions of varying levels of knowledge while critically merging the learner's knowledge with disciplinary insights and personal experiences. Assessment should include learner reflection on their goals and accomplishments and avoid overly narrow measurement of skills or knowledge without a complete context of interpretation. While this description is not that different from the literature that is current, it does provide, I believe, the beginning constructivist teacher or designer with the elements of a 'blueprint' that is transferable to and consistent with the creation of a learning environment sympathetic to the intentions of a constructivist philosophy. When asked specifically to comment on the learning activities (often cited in the literature as constructivist), the participant-experts could only arrive at consensus on one activity considered essential to constructivism and that was the gathering of resources around a topic under development within content. Creative works, problem-based learning, case based learning along with asynchronous discussions and portfolios/e-portfolios were close to consensus but did not achieve the 80% agreed upon demarcation. What remains missing is consensus on the activities that have been described in the literature as promoting or encouraging a constructivist environment or classroom. In attempting to reach consensus, I was left with the feeling that it was the core concept of constructivism and the role of the instructor which captured the imagination of the participant-experts and that they felt that many activities, when activated in this core context, could be construed as constructivist or certainly modified to support the core 179 constructivist description offered by this study. This was borne out by some of the participant-discussion that took place in Round 4. So why did the participant-experts remain in this 'broad view' of constructivism and not narrow down into a more 'specific view'? Why did each time that they looked at specifics it was 'dismissed' as being more than constructivist? It is my opinion that they were observing a dynamic which I have seen in my own teaching and design work. This dynamic suggests that while a particular activity or learning event might be constructivist there are other elements that impact on the activity, the influence of the instructor, the design of the interaction, the context of the content, the needs of the learners, which all contribute to the process and which do not necessarily make the approach inherently constructivist. This hesitation to call an event or activity 'constructivist' remains limited by our inability to have a 'more robust conceptualization of constructivism'. While intriguing, this component of the study, consensus on specific activities construed as constructivist, will have to be left to future research to discover that consensus. What is interesting about the study's findings is that with respect to constructivism, while tied to online learning in much of the literature, it remains a theory that eludes formal structure. It appears to be an internalization that can be encouraged consciously though the design of the learning environment and active participation on the part of the learners, and the instructor. Philosophy or Theory? Another aspect of this study, which remains a mystery, as there was no clear consensus on the issue, is the debate over constructivism as theory versus philosophy. 180 This debate has taken place in the literature and did emerge in the discussions amongst the participant-experts. I was left wondering whether constructivism was a philosophy or a theory of teaching (or of learning) and also left with the overriding question of what is the difference between a philosophy and a theory in the context of influencing practice? In truth, the data gathered by this study does not come down on either side, so the debate continues to be available for study by future researchers. However, based on what I have personally learned in this study, coupled with my own experiences and what the 'experts' say in the literature, I think it is reasonable to think of constructivism as a philosophy or 'global view', which informs the teacher and the student, as they teach and learn. This philosophy then forms the basis for their thinking and actions within the learning environment no matter what learning theory is employed. 181 REFERENCES Ackoff, R.A. (1974). Redesigning the future: a Systems approach to societal problems. Wiley: New York. Ackoff, R.A. (1979). The future of Operational Research is past. Journal of the Operational Research Society 30(2): 93-104. Adams, L. , Toomey, L. & Churchill, E. (1999). Distributed Research Teams: Meeting Asynchronously in Virtual Space. Journal of Computer Mediated Communication. 4(4). Retrieved July 28, 2003 from http://jcmc.huji.ac.il/vol4/issue4/adams.html Airasian, P. W., & Walsh, M . E. (1997). Constructivist cautions. Phi Delta Kappan, 78, 444-449. Aliponga, J. (2003). Developing fluency through questioning strategies. Paper in the 2003 Pan - SIG Conference Proceedings. Retrieved February 20, 2004 from http://www.jalt.org/pansig/2003/HTML/Aliponga.htm Allin Corporation. (2002). Healthcare Consulting Group - Nominal Group Technique Survey Application. Case Study Report retrieved July 24, 2003 from http://www. all in.com/Consulting/case_studies/HeaJthcare%20Consulting%20Gro up.pdf Ally, M . (2004). Foundations of educational theory for online learning. In Terry Anderson & Fathi Elloumi (Eds.), Theory and practice of online learning. Athabasca, A B : Athabasca University. Retrieved February 5, 2005 from http://cde.athabascau.ca/online_book/index.html Anderson, T. & Kanuka, H. (2003). E-Research: Methods, Strategies and Issues. Boston: M A . Pearson Education, Inc. Archer, N . P. (1989). Improving idea generation during decision making in small group computer conferences. Canadian Journal of Educational Communication 18(2), 111-123. Archer, N . P. (1990). A comparison of computer conferences with face-to-face meetings for small group business decisions. Behaviour and Information Technology 9(4), 307-317. Australian Flexible Learning Framework for the National Vocational Education and Training System 2000-2004. (2003). Delivering Learning Online: Asynchronous Environments. Retrieved July 24, 2003 from http://www.tafe.net/learnscope/dlo/communications5.html Baines, L . A. , & Stanley, G. (2000). We want to see the teacher: Constructivism and the rage against expertise. Phi Delta Kappan, 82(4), 327-330. Barell, J. (1991). Teaching for thoughtfulness: Classroom strategies to enhance intellectual development. New York: Longman. Barrett-Lennard, G.T. (1998). Carl Roger's helping system: Journey and substance. London: Sage. Batagelj, Z., & Vehovar, V . (1998). Technical and methodological issues in www surveys, St. Louis, MO. Berger, C. & Kam, R. (1996). Definitions of instructional design. Online resource adapted from Training and Instructional Design, Applied Research Laboratory, 182 Penn State University. Retrieved September 21, 2003 from http://www.umich.edu/~ed626/define.html Beruvides, M . G . (1995). Group decision support systems and consensus building: Issues in electronic media, Computers & Industrial Engineering, 29(1-4). 601-605. Retrieved July 27, 2003 from ScienceDirect Database, UBC Library. Beruvides, M . G. (1999). A Nominal Group Technique Analysis of the Management of Technology in Developing Nations. Presentation to Ancient Civilizations and Modern Technology -1AMOT Conference, 15-17 March, Cairo, Egypt. Bopry, J. (1999). The warrant for constructivist practice within educational technology. Educational Technology Research & Development, 47 (4), 5-26. Bonk, C , & Cunningham, D.J. (1998). Searching for learner-centered, constructivist, and sociocultural components of collaborative educational learning tools. In C. Bonk & K. King (Eds.), Electronic collaborators: Learner-centered technologies for literacy, apprenticeship, and discourse . Mahwah, NJ: Lawrence Erlbaum Associates. Booth, S., Struck, C , & Thomas, C. (1986). Our reform experiment is turning knowledge into results. The Executive Educator, 5(4), 32-33. Boudourides, M . A . (2003). Constructivism, Education, Science, and Technology, Canadian Journal of Learning and Technology, 29(3). 5-20. Retrieved September 10, 2004 from http://www.cilt.ca/content/vol29.3/cilt29-3_aitl .html Boudreau, P (2000). L'expertise d'un enseignant associe, McGill Journal of Education 55(1), 53-70. [English abstract only] Bramwell, L . & Hykawy, E. (1999). The Delphi Technique: A possible tool for predicting future events in nursing education. Canadian Journal of Nursing Research 30(4): 47-58. Bransford, J.D., Brown, A . L . & Cocking, R.R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Committee on Developments in the Science of Learning, Commission on Behavioral and Social Sciences and Education, National Research Council. Retrieved April 16, 2005 from http://books.nap.edu/html/howpeoplel/ Brooks, J. G., & Brooks, M . G. (1993). In search of understanding: The case for constructivist classrooms. Alexandria, V A : Association for Supervision and Curriculum Development. Brooks, M . G., & Brooks, J. G. (1999). In search of understanding: The case for constructivist classrooms (2nd ed.). Alexandria, V A : Association for Supervision and Curriculum Development. Brown, T. (1997). Multimedia in Education -Conclusions. Retrieved online September 27, 1999 from http://l29.180.87.4/Units/CurricSt/CSIT513/573/573 12.html Bruffee, K . (1999). Collaborative learning: Higher education, interdependence, and the authority of knowledge. Baltimore: Johns Hopkins University Press. Bruner, J. S. (1966). Toward a theory of instruction. In J. Hartley & I. K. Davies (Eds.), (1978). Contributions to an educational technology, Vol. 2. London: Kogan Page. Bruner, J.S. (1983). Education as a social invention. Journal of Social Issues (39), 129-141. Bull, K . S., Montgomery, D. L . & Kimball, S. L . (1999) Designing instructional content: Constructivist Instructional Design. In Oklahoma State University, Quality 183 University Instruction: A Teaching Effectiveness Training Program, Stillwater, OK, Oklahoma State University. Retrieved February 12, 2005 from: http://home.okstate.edu/homepages.nsf/toc/EPSY5213Reading3b Bull, S., & McFarlane, M . (2000). Soliciting Sex on the Internet: What Are the Risks for Sexually Transmitted Diseases and HIV? Sexually Transmitted Diseases, 27(9), 545-550. Butler, C. & Seiler, C. (1984). Hegel: The letters. Bloomington: Indiana University Press. Center for Rural Studies. (1998). Guidelines for using the Nominal Group Technique. Retrieved July 8, 2003 from http://crs.uvm.edU/gopher/nerl/group/a/iTieet/Exercise7/b.html Chappie, M . & Murphy, R. (1996). The nominal group technique: Extending the evaluation of students' teaching and learning. Assessment & Evaluation in Higher Education. 21(2). 147-160. Retrieved July 8, 2003 from Academic Search Premier Database, U B C Library. Chew, R. (1996). Jean-Jacques Rousseau. Lucidcafe. Retrieved July 14, 2005 from http://www2.lucidcafe.com/lucidcafe/libraiy/96iun/rousseau.html Clayton, M.J. (1997) Delphi: A technique to harness expert opinion for critical decision-making tasks in education, Educational Psychology, 17. 273-287. Clements, D. H. (1997). (Mis?)Constructing constructivism. Teaching Children Mathematics, 4(4), December, 3 pages. Available at: Galileo Periodical Abstracts. Cobern, W. (1993). Contextual Constructivism: The impact of culture on the learning and teaching of science. In: K . Tobin (Ed) The Practice of Constructivism in Science Education, pp 51-69. NJ: Lawrence-Erlbaum, Hillsdale. Cohen, J. (1996). Computer mediated communication and publication productivity among faculty. Internet Research: Electronic Networking Applications and Policy, 6(2/3), 41-63. Colomb, G. G., & Simutis, J. A . (1996). Visible conversation and academic inquiry. In S. C. Herring (Ed.), Computer-mediated communication: Linguistic, social and cross-cultural perspectives (pp. 203-224). Amsterdam: John Benjamins Publishing. Coomber, R. (1997). Using the internet for survey research. Sociological Research Online, 2(2). Costelloe, T. (2003). Giambattista Vico. The Stanford Encyclopedia of Philosophy (Summer 2003 Edition), Edward N . Zalta (ed.), Retrieved July 14, 2005 from http://plato.stanford.edu/archives/sum2003/entries/vico/ Couper, M . P. (2001). Web surveys: A review of issues and approaches. The Public Opinion Quarterly, 64(4.), 464-494. Crawford, K. (1996). Vygotskian approaches to human development in the information era. Educational Studies in Mathematics (31), 43-62. Cresswell, J.W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2 n d Ed). Thousand Oaks: CA. Sage Publications. Crocco, M.S. (2001). Leveraging constructivist learning in the social studies classroom. Contemporary Issues in Technology and Teacher Education [Online serial], 7(1), Retrieved April 16, 2005 from http://www.citejournal.org/voll/iss3/currentissues/socialstudies/article2.htm Cunningham, D., Duffy, T. M . , & Knuth, R. (1993). Textbook of the future. In C. 184 McKnight (Ed.), Hypertext: A psychological perspective. London: Ellis Horwood Publishing. Curry, L . (1990). A Critique of the Research on Learning Styles. Educational Leadership 48 (2), 50-55. Dalgarno, B. (2001). Interpretations of constructivism and consequences for Computer Assisted Learning. British Journal of Educational Technology, 32(2). 183-194. Dalkey, N . (1972) Studies In the Quality of Life. Lexington, M A : Lexington Books. Danielson, C. (1996). Enhancing professional practice: A framework for teaching. Alexandria, V A : Association for Supervision and Curriculum Development. Delbecq, A . L . & Vande Ven, A . H . (1971). A group process model for problem identification and program planning. Journal of Applied Behavioral Science. VII, 466-91. Delbecq, A . L . , Vande Ven, A . H . & Gustafson, D.H. (1975). Group Techniques for Program Planners. Glenview, Illinois: Scott Foresman and Company, de Lisle, Peter (1997). What is Instructional design theory? Retrieved September 21, 2003 from http://hagar.up.ac.za/catts/learner/peterdl/ID%20Theory.htm Denzin, N . & Lincoln, Y . (Eds) (2003). Strategies of Qualitative Inquiry (2 n d Ed). Thousand Oaks: CA. Sage Publications. Dewey, J. (1897). My pedagogic creed. The School Journal (LIV)3, p. 77-80. Retrieved July 14, 2005 from http://www.infed.org/archives/e-texts/e-dew-pc.htm Dewey, J. (1916) Democracy and education, New York: Macmillan. Dewey, J. (1938). Experience and education. New York: Macmillan. Dobozy, E. (1999). Constructivist and Montessorian perspectives on student autonomy and freedom. Proceedings Western Australian Institute for Educational Research Forum. Retrieved February 12, 2005 from http://education.curtin.edu.au/waier/forums/1999/dobozy.html Dougiamas, M . (1998). A journey into constructivism. Retrieved February 20, 2005 from http://dougiamas.eom/writing/constructivism.html#conclusions Doolittle, P.E. (1999). Constructivism and online education, [online paper]. Virginia Polytechic Institute & State University. Retrieved March 13, 2002 from http://www.tandl.vt.edu/doolittle/research/tohel999/pedagogy.html Doolittle, P.E. (1999a). Constructivism: The career and technical education perspective. Journal of Vocational and Technical Education, I6(\). Retrieved April 16, 2005 from http://scholar.lib.vt.edu/ejournals/JVTE/vl6nl/doolittle.html Dowling, K . L . and St. Louis, R.D. (2000). Asynchronous implementation of the nominal group technique: Is it effective? Decision Support Systems, 29. 229-248. Retrieved July 27, 2003 from ScienceDirect Database, UBC Library. Driscoll, M . P. (2000). Psychology of learning for instruction (2nd ed.). Needham Heights, M A : Allyn and Bacon. Duffy, T. M . & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery of instruction. In D. H . Jonassen (Ed.), Handbook of Research for Educational Communications and Technology (170-198). New York: Simon Schuster Macmillan. Duggan, S & Cox, J (1999). Evaluating one-to-one teaching in general practice, Medical Teacher 27(1), 61-66. Dunham, R.B. (1998). Nominal Group Technique: A Users' Guide. Retrieved July 8, 185 2003 from http://instruction.bus.wisc.edu/obdemo/readings/ngt.html Emery, M . & Purser, R.E. (1996). The Search Conference. San Francisco: Jossey-Bass. Feltovich, P.J., Spiro, R.J., Coulson, R.L., & Feltovich, J. (1996). Collaboration within and among minds: Mastering complexity, individually and in groups. In T. Koschman (Ed.), Computer systems for collaborative learning (25-44). Hillsdale, NJ: Lawrence Erlbaum. Fish, L.S. & Busby, D . M . (1996). The Delphi Method, In: D.H. Sprenkle & S.M. Spoon (Eds.). Research Methods In Family Therapy. New York: Guilford Press. Fosnot, C. T. (1989). Enquiring teachers, enquiring learners. New York: Teachers College Press. Fosnot, C. T. (1992). Constructing constructivism. In T. M . Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 167-176). Hillsdale, NJ: Lawrence Erlbaum Associates. Fosnot, C. T. (Ed.). (1996). Constructivism: Theory, perspectives, and practice. New York: Teachers' College Press, Columbia University. Foster, G. (1994). Fishing with the Net for research data. British Journal of Educational Technology, 25(2), 91-97. Fricker, R. D., & Schonlau, M . (2002). Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods, 14(4), 347-365. Fuller, I., Gaskin, S., & Scott, I. (2003). Student perceptions of geography and environmental science fieldwork in the light of restricted access to the field, caused by foot and mouth disease in the U K in 2001. Journal of Geography in Higher Education. 27(1). 79-102. Retrieved July 15, 2003 from EBSCO Database, UBC Library. Gabriel, M . (2004). Learning together: Exploring group interactions online. Journal of Distance Education, 19(\). 54-72. Gabriel, M . , Ostridge, R., & Doiron, S. (2003). Exploring better practices in online teaching and learning: Determining the perspectives of instructors and learners in a Delphi Process. Paper presented at CADE, St. John's, N L . June 7-11. Gagne, R. M . & Dick, W. (1983). Instructional psychology. Annual Review of Psychology, 34, 261-295. Galagher, J., Sproull, L. , & Kiesler, S. (1998). Legitimacy, Authority, and Community in Electronic Support Groups, Written Communication, 15(4), Oct 98. 493-530. Sage Publications Garrison, R. (1997). Computer conference: the post-industrial age of distance education. Open Learning, 12(2). 5, 6 Gepson, J., Marinko, M.J. , & Belina, J. (1981). Nominal Group Techniques. Training and Development Journal. September. Retrieved July 8, 2003 from EBSCO Database, U B C Library. Gerdes, K. E., & Benson, R. A . (1995). Problems of inner-city schoolchildren: Needs assessment by Nominal Group Process. Social Work in Education, 17, 139-147. Gibson, F. & Soanes, L . (2000). The development of clinical competencies for the use on a paediatric oncology nursing course using a nominal group technique. Journal of Clinical Nursing. 9. 459-469. Retrieved July 15, 2003 from EBSCO Database, U B C Library. 186 Glasersfeld, E. v. (1984). Radical constructivism. In P. Watzlawick (Ed.), The invented reality. Cambridge, M A : Harvard University Press. Glasersfeld, E. v. (1988). The reluctance to change a way of thinking. The Irish Journal of Psychology 9(1), 83-90. Glasersfeld, E. v. (1990). Environment and communication. In L. P. Steffe & T. Woods (Eds.), Transforming children's mathematical education. Hillsdale, NJ: Erlbaum. Glasersfeld, E. v. (1993). Questions and answers about radical constructivism. In K. Tobin (ed.). The practice of constructivism in science education. Hillsdale, NJ: Lawrence Erlbaum Associates. Glasersfeld, E.v. (1998). Von Glasersfeld's answers - November 1998. Retrieved July 13, 2005 from http://www.oikos.org/vonanswernov.htm Glasersfeld, E.v. (1999). Von Glasersfeld's answers - July 1999. Retrieved July 13, 2005 from http://www.oikos.org/vonansweriul.htm Glasersfeld, E.v. (2000). Von Glasersfeld's answers - April 2000. Retrieved July 13, 2005 from http://www.oikos.org/vonansaprOQ.htm Glasersfeld, E.v. (2001). Von Glasersfeld's answers- October 2001. Retrieved July 13, 2005 from http://www.oikos.org/vonansoct01.htm Glasersfeld, E.v. (2001a). Von Glasersfeld's answers - December 2001. Retrieved July 13, 2005 from http://www.oikos.org/vonansdecO 1 .htm Glasersfeld, E.v. (2002). Von Glasersfeld's answers - October 2002. Retrieved July 13, 2005 from http://www.oikos.org/vonansoct02.htm Glasersfeld, E.v. (2002a). Von Glasersfeld's answers - December 2002. Retrieved July 13, 2005 from http://www.oikos.org/vonansdic02.htm Glasersfeld, E.v. (2005). Von Glasersfeld's answers - May 2005. Retrieved July 13, 2005 from http://www.oikos.org/vonansmag05.htm Goldman, J. D. G. & Hocking, C. G. (1999). Internet educational usage by High School students around the world. Educational Practice and Theory, 27(1). 67-85 Goldman, J.D.G & Torrisi-Steele, G. (2002). Constructivist pedagogies of interactivity on a C D - R O M to enhance academic learning at a tertiary institution. International Journal of Educational Technology. 5(1). Goodman, N . (1984). Of mind and other matters. Cambridge, M A : Harvard University Press. Goodman, R. (2003). William James. The Stanford Encyclopedia of Philosophy (Spring 2003 Edition), Edward N . Zalta (ed.). Retrieved July 14, 2005 from http://plato.stanford.edu/archives/spr2003/entries/iames/ Gordon, S. (2003). Methods to solicit and aggregate expert opinions - Nominal Group Technique. Retrieved July 8, 2003 from http://oregonstate.edu/~gordons/researclVprocesses/ngt.htm Gulati, S. (2004). Constructivism and emerging online learning pedagogy: a discussion for formal to acknowledge and promote the informal. Paper presented at the Annual Conference of the Universities Association for Continuing Education -Regional Futures: Formal and Informal Learning Perspectives, Centre for Lifelong Learning, University of Glamorgan, 5-7 April 2004. Retrieved April 16, 2005 from http://www.leeds.ac.uk/educol/documents/00003562.htm Harasim, L. (1991). Researching online education: Perspectives and methodologies. Paper presented at the A E R A , Chicago, IL 187 Hart, G. (Ed.) (2000). Readings and Resources in Global Online Education. Melbourne: Whirligig Press. Hartley, J. & I. K. Davies. (Eds.), (1978). Contributions to an educational technology, Vol. 2. London: Kogan Page. Harvey, J. (1998). Working with groups of staff and students. LTDI Evaluation Cookbook. Learning Technology Dissemination Initiative. Web version first published Feb 1999. Retrieved July 8, 2003 from http://wvvw.icbl.hw.ac.uk/ltdi/cookbook/infb\vorking_w-ith_groups/index.html#e ndhead Hausfather, S. (1996). Vygotsky and schooling: Creating a social context for learning. Action in Teacher Education (18), 1-10 Hausfather, S. (2001). Where's the content? The role of content in constructivist teacher education. Educational Horizons (Fall), 15-19 Hegel, G. W. (1807/1949). The phenomenology of mind (J.B. Baillie, Trans.). London: Allen Unwin. Hegel, G.W. (1977). Phenomenology of spirit. Oxford University Press. Henriques, L. (1997). Constructivist teaching and learning. Excerpt from: A study to define and verify a model of interactive-constructive elementary school science teaching. Unpublished Ph.D. Dissertation, University of Iowa, Iowa City, IA. Hinds, P., & Kiesler, S. (Ed.). (2002). Distributed work. Cambridge, M A : MIT Press. Hodson, D., & Hodson, J. (1998). From constructivism to social constructivism: A Vyotskian perspective on teaching and learning science. School Science Review, 79, 33-41. Hoffmann, K. (2003). GMP Survey - First Letter. Personal email correspondence to D.P. Janes. Received July 27, 2003. Hoffmann, K. , Schelle, H . & Yalpani, A . (2002). The future of project management: Results of an International expert survey. Presentation to the 16th World Congress of the IP MA (International Project Management Association) in Berlin (4-6 June 2002). Retrieved July 24, 2003 from http://www.teamspring.net/S=7576adab219a6cb05ef46ff230ee9525/asynchron/ Honebein, P. (1996). Seven goals for the design of constructivist learning environments. In B. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design (pp. 11-24). New Jersey: Educational Technology Publications. Honebein, P.C., Duffy, T.M., Fishman, B.J. (1993). Constructivism and the design of learning environments: Context and authentic activities for learning, Designing Environments for Constructive Learning. Berlin: Springer-Verlag. 87-108. Hooper, S., & Hannafin, M . J. (1991). The effects of group composition on achievement, interaction, and learning efficiency during computer-based cooperative instruction. Educational Technology Research and Development, 39(3), 27-40. Hoyle, J. R. (1978). Teacher versus administrator: A growing crisis. Planning & Changing, 9. 203-209. Hsiao, J. W. D. L. (1996). CSCL Theories. Retrieved July 8, 2003 from http://www.edb.utexas.edU/csclstudent/dhsiao/theories.html#what Innes, J.E. (1999). Evaluating consensus building. In Susskind, L. , McKearnan, S., & Thomas-Larmer, J. (Eds.), The Consensus Building Handbook: A comprehensive 188 guide to reaching agreement. Thousand Oaks, C A : Sage. 647-654. Iowa State University Extension. (2001). Tips for Nominal Group Process. Retrieved July 8, 2003 from http://www.extension.iastate.edu/communities/tools/dec-isions/nominal.html James, W. (1899). Talks to teachers. Cambridge, Mass. Retrieved July 15, 2005 from http://www.des.emory.edu/mfp/ttpreface.html Janes, D. P. (2003). Personal email correspondence with Professor H . Schelle. July 24, 2003. Janes, D. P. (2003a). Personal email correspondence with Dr. M . Beruvides. July 25, 2003. Janes, D. P. (2003b). Personal email correspondence with Dr. K. Hoffmann. July 26 and 27, 2003. Janes, D. P. (2003c). Personal email correspondence with Dr. M . Gabriel, November 18, 2003. Janes, D. P. (2004). Personal telephone conversation with J. Rosario, January, 2004. Janes, D. P. (2004a). Personal email correspondence with M . Boucher, February 10, 2004. Janes, D. P. (2004b). Personal email and telephone correspondence with A . Doree, January-March, 2004. Janes, D.P. (2004c). Personal email and telephone correspondence with Dr. Tom Sork, Spring, 2004. Johnson, D. W., & Johnson, R. T. (1996). Cooperation and the use of technology. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 170-198). New York: Simon & Schuster Macmillan. Johnson, R.A. & Richards, R.R. (1983). The use of the nominal group technique to isolate teacher job stressors, Children and Youth Services Review, 5(3). 289-300. Retrieved July 27, 2003 from ScienceDirect Database, U B C Library. Johnston, J. (1984). Evaluating new information technologies. San Francisco, C A : Jossey-Bass. Jones, S. (2004). Using the nominal group technique to select the most appropriate topics for postgraduate research students' seminars. Journal of University Teaching and Learning Practice 20-34. Retrieved February 5, 2005 from http://jutlp.uow.edu.au/ Jones, J. & Hunter, D. (1999). Using the Delphi and nominal group technique in health services research. Chapter 5 in C. Pope & N . Mays (Eds) Qualitative Research in Health Care (2 n d Ed.) B M J Publishing Group. Retrieved July 8, 2003 from http ://bmi pg. co m/qrhc/chapter 5 .htm 1 Jonasson, D.H. (n.d.). Thinking technology: Toward a constructivist design model. Retrieved February 12, 2005 from http://web.archive.Org/web/l 9990427173629/http://ouray .cudenver.edu/~slsanfor/ cnstdm.txt Jonassen, D. H. (1991a). Evaluating constructivistic learning. Educational Technology, 31(9), 28-33. Jonassen, D. H. (1991b). Objectivism versus constructivism: Do we need a new philosophical paradigm? ETR&D, 39(3), 5-14. 189 Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M . Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory (Vol. 2, pp. 215-239). New Jersey: Lawrence Erlbaum Associates. Jonassen, D. H. & McAleese, T.M.R. (n.d.). A Manifesto for a constructivist approach to technology in higher education. Retrieved February 12, 2005 from: http://web. archive. orgAveb/20000302162530/http://www.icbl.hw.ac.uk/ctl/msc/ce ejwl/paper 11 .html Joppe, M . (n.d. 1) The Research Process: The Nominal Group Technique. Retrieved July 8, 2003 from http://www.tTerson.ca/~mjoppe/ResearchProcess/841TheNominalGroupTechniqu e.htm Joppe, M . (n.d. 2). The Research Process: The Delphi Method. Retrieved July 8, 2003 from http://www.ryerson.ca/~mjoppe/ResearchProcess/84rrheDelphiMethod.htm Kaminski, S. (2003). Commentary on Vico's Ingenium. Retrieved July 15, 2005 from http://www.shkaminski.com/Classes/Readings/Ingenium.htm Kant, E. (1781/1946). Critique of pure reason (J. M . D. Meiklejohn, Trans.). New York: Dutton. Kant, E. (1983). Perpetual peace and other essays on politics and morals. Indianapolis, ID: Hackett Publishing Company. Kant, I. (2001). The Columbia Encyclopedia, 6th ed. New York: Columbia University Press, 2001-04. Retrieved July 14, 2005 from http://www.bartleby.com/65/ka/Kant-lmm.html Kapiriri, L. , Robbestad, B., & Frithjof Norheim, O. (2003). The relationship between prevention of mother to child transmission of HIV and stakeholder decision making in Uganda: implications for health policy. Health Policy. (In Press). Retrieved July 27, 2003 from ScienceDirect Database, UBC Library. Kehoe, C. M . , & Pitkow, J. E. (1996). Surveying the Territory: GVU's Five WWW User Surveys. World Wide Web Journal, 7(3). Kincheloe, J. L. , & McLaren, P. L . (1994). Rethinking critical theory and qualitative research. In N . K. Denzin & Y . S. Lincoln (Eds.), Handbook of Qualitative Research. London: Sage Kingsley, P., & Anderson, T. (1998). Facing life without the Internet. Internet Research: Electronic Networking Applications and Policy, 8(4), 303-312. Kirschenbaum, H . & Henderson, V . L . (Ed.). (1990). The Carl Rogers reader. London: Constable. Kraut, R., Olson, J., Banaji, M . , Bruckman, A. , Cohen, J. & Couper, M . (2003). Psychological Research Online: Opportunities and Challenges. Retrieved February 5, 2005 at http://www-2.cs.cmu.edu/~kraut/RKraut.site.files/articles/kraut03-PsychologicalResearchOnline.pdf Kreber, C. (2002). Controversy and consensus on the scholarship of teaching. Studies in Higher Education, 27(2). 151-167. Larochelle, M . , Bednarz, N . , & Garrison, J. (Eds.). (1998). Constructivism and education. Cambridge, U K : Cambridge University Press. 190 Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press. Lilly, I. (1967). Friedrich Froebel: A selection from his writings, 'The Education of Man'. London, England: Cambridge University Press. Linstone, H.A. & Turoff, M . (1975) The Delphi Method: techniques and applications. Reading, M A : Addison-Wesley Lloyd-Jones, G., Fowell, S. & Bligh, J.G. (1999). The use of the nominal group technique as an evaluative tool in medical undergraduate education. Medical Education. 33. 8-13. Lomax, P. & McLeman, P. (1984). The uses and abuses of nominal group technique in polytechnic course evaluation. Studies in Higher Education, 9(2). 183-190. Luppicini, R. (2003). Towards A Cyber-Constructivist Perspective (CCP) of educational design. Canadian Journal of Learning and Technology. 29(1), 5-17. Maani, K . E. (2002). Consensus building through systems thinking: The case of policy and planning in healthcare. The Australian Journal of Information Systems, 9(2). Retrieved February 13, 2005 from http://www.uow.edu;au/ajis/vol92p9.html Mackenzie, M . (1909). Hegel's educational theory and practice. London: Swan Sonnenshein. Mann, C , & Stewart, F. (2000). Internet communication and qualitative research: A handbook for researching online. Thousand Oaks, C A : Sage Publications. Manwaring, G. (1998). Nominal Group Technique. LTDIEvaluation Cookbook. Learning Technology Dissemination Initiative. Web version first published Feb 1999. Retrieved July 8, 2003 from http://vvww.icbl.hw.ac.Uk/ltdi/cookbook/nominal_group_technique/index.html#en dhead Marton, F., Hounsell, D., and Entwistle, N . (1997). The Experience of Learning. Implications for Teaching and Studying in Higher Education (2nd ed.), Edinburgh: Scottish Academic Press. Matthews, M . R. (1992). Old wine in new bottles: A problem with constructivist epistemology. Philosophy of Education Society. Retrieved November 17, 1999 from: http://www.ed.uiuc.edu/EPS/PES-Yearbook/92_docs/Matthews.htm McCarthey, S. J., & Raphael, T. E. (1992). Alternative research perspectives. In J. W. Irwin & M . A . Doyle (Eds.), Reading/Writing Connections: Learning: from research (pp. 2-30). Newark, NJ: IRA. McCormick, M . (2005). Immanuel Kant (1724-1804) Metaphysics. The Internet Encyclopedia of Philosophy. Retrieved July 14, 2005 from http://wwv\^.iep.utm.edu/k/kantmeta.htm McGriff, S.J. (2001). ISD Knowledge base/Theoretical introduction. Retrieved September 8, 2003 from http://www.personal.psu.edu/faculty/s/j/sjm256/portfolio/kbase/Theories&Models /theoryintro.html#idtheory McKill ip, J. (1987). Need Analysis: Tools for the Human Services and Education. Newbury Park, C A : Sage Publications. McMahon, M . (1997, December 7-10). Social constructivism and the world wide web - A paradigm for learning. Paper presented at the ASCILITE '97, Perth, Australia. 191 McMurray, A.R. (1994). Three decision-making aids: Brainstorming, nominal group technique and Delphi technique. Journal of Nursing Staff Development. 10. 62-65. Mergel, B. (1998). Instructional design & learning theory. Unpublished graduate paper for Educational Communications and Technology, University of Saskatchewan, May, 1998. Retrieved February 12, 2005 from http://www.usask.ca/education/coursework/802papers/mergel/mergel.PDF Middleton, J. A . (2000). From manuscript to multimedia: How technologies transform education research. Current Issues in Education, 3(2). Miller, J.B. (2002). Examining the interplay between constructivism and different learning styles. ICOTS6: The Sixth International Conference on Teaching Statistics, Cape Town, South Africa. July 7-12, 2002. Retrieved February 12, 2005 from http://www.stat.auckland.ac.nz/~iase/publications/l/8a4_mill.pdf MIT-Harvard Public Disputes Program (n.d.). The Program on Negotiation at Harvard Law School's Short Guide to Consensus Building. Retrieved September 10, 2004 from http://vveb.mit.edu/publicdisputes/practice/ Moallem, M . (2001). Applying constructivist and objectivist learning theories in the design of a web-based course: Implications for practice. Educational Technology & Society, 4(3), 1436-4522. Mohtessori, M . (1946/1963). Education for New World. Madras, India: Kalakshetra Publications. Morrison, D. (2004). A study of holistic thinking in an agricultural leadership development program using asynchronous computer conference. Ph.D. Dissertation, OI SE, University of Toronto. Morrison, J.L. (1998). The nominal group process s an instructional tool. The Technology Source. June. Retrieved July 8, 2003 from http://ts.mivu.org/default.asp?show-article&id:=466 Murphy, E. (1997a). Constructivism from philosophy to practice, [website, html pages]. Elizabeth Murphy. Retrieved March 12, 2002 from: http://www.stemnet.nf.ca/~elmurphy/emurphy/cle.html Murphy, E. (1997b). Constructivist learning environments, [www html page]. Retrieved March 12, 2002 from: http://www.stemnet.nf.ca/~elmurphy/emurphy/constructivism.html Murphy, K. L. , & Cifuentes, L. (2001). Using Web tools, collaborating, and learning online. Distance Education, 22(2), 285-305. Neiger, B.L. , Barnes, M.D. , Thackeray, R., & Lindman, N . (2001). Use of the Delphi Method and Nominal Group Technique in front-end market segmentation. American Journal of Health Studies. 17(3). 111-119. Retrieved July 15, 2003 from EBSCO Database, UBC Library. Nelson, J. S., Jayanthi, M . , Brittain, C. S., Epstein, M . H., Bursuck, W. D. (2002). Using the Nominal Group Technique for homework communication decisions. Remedial & Special Education. 23(6), p. 379-387. Nosek, B. A. , Banaji, M . , & Greenwald, A . G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics, 6(\), 101-115. O'Brien, R. (1998). An overview of the methodological approach of Action Research. Retrieved February 13, 2005 from http://www.web.net/~robrien/papers/arfinal.html 192 Ohio State University Extension. (2000). Building dynamic groups: Nominal Group Technique. Facilitation #6. Retrieved July 8, 2003 from http://www.ag.ohio-state. edu/~bd g/pd f_doc s/d/F06. pdf Orem, R.C. (1971). Montessori Today. New York, N Y : Capricorn Books. Orlikowski, W. J. (2000). Using technology and constituting structures: A practice lens for studying technology in organizations. Organizational Science, 77(4), 404-428. Palloff, R. M . , & Pratt, K. (1999). Building learning communities in cyberspace. San Francisco: Jossey-Bass. Passmore, J. (1967). Philosophy. In Edwards (Ed.). The encyclopedia of philosophy. 6, p. 216-226. Pazos, M . , Perazzoli, V. , Jiang, J., Canto., A . , & Beruvides, M . (2001). Application of the Nominal Group Technique in an Internet versus a traditional environment. In Proceedings of the HE Annual Conference, Pittsburgh, University of Pittsburgh. Perkins, D. (1999). The many faces of constructivism. Educational Leadership, 57(3), 6 -11. Perkins, D. N . (1991). Technology meets constructivism: Do they make a marriage? Educational Technology, May, 18-23. Pfeiffer, J. (1969). New Look at Education. Princeton, NJ: Western. Phillips, D. C. (1995). The good, the bad, and the ugly: The many faces of constructivism. Educational Researcher, 24, 5-12. Phahlamohlaka, J. & Friend, J. (2003). Community planning for rural education in South Africa. European Journal of Operational Research. (In Press). Retrieved July 27, 2003 from ScienceDirect Database, U B C Library. Piaget, J. (1977). The Development of Thought: Equilibration of Cognitive Structures. New York: Viking. Pintrich, P. (2000). The role of goal orientation in self-regulated learning. In M . BOekaerts, P. Pintrich, & M . Zeidner (Eds.) Handbook of self-regulation (pp. 452-494). San Diego, CA: Academic Press. Pithers, R. & Soden, R. (2000). Critical thinking in education: A review, Educational Research, 42(3), 237-249. Prawat, R. S. (1996). Constructivisms, modern and postmodern. Educational Psychologist, 31(3/4), 215-225. Prawat, R. S., & Floden, R. E. (1994). Philosophical perspectives on constructivist views oflearning. Educational Psychology, 29,37-48. Prosser, M . and Trigwell, K. (1999). Understanding Learning and Teaching. The Experience in Higher Education. Buckingham: The Society for Research into Higher Education and the Open University. Redding, P. (2002). Georg Wilhelm Friedrich Hegel. The Stanford Encyclopedia of Philosophy (Summer 2002 Edition), Edward N . Zalta (ed.). Retrieved July 14, 2005 from http://plato.stanford.edu/archives/sum2002/entries/hegel/ Reigeluth, C M . (1997). Instructional theory, practitioner needs, and new directions: Some reflections. Educational Technology, (37)\, 42-47 Reigeluth, C M . , (1999). What is instructional-design theory, and how is it changing? In C M . Reigeluth (Ed.), Instructional design theories and models: A new paradigm of instructional theory, volume 11. Hillsdale, NJ: Lawrence Erlbaum Associates. pp. 425-459. 193 Reusser, K. (2002). Co-constructivism in educational theory and practice. International Encyclopedia of the Social & Behavioral Sciences, 2058-2062. Retrieved September 25, 2005 from ScienceDirect Database, UBC Library. Reynolds, C. G. (1997). Dimensions of capacity for shared decision making. Unpublished doctoral dissertation; Georgia Southern University. Rezabek, R. J. (2000). Online focus groups: Electronic discussions for research. Forum: Qualitative Sozial Forschung/Forum: Qualitative Social Research (FQS), 7(1). Rheingold, H. (1998). The Virtual Community, [online book]. Retrieved April 21, 2002 from http://www.rheingold.com/vc/book/ Richardson, V . (Ed.). (1997). Constructivist teacher education: Building new understandings. Washington, D.C.: Falmer Press. Rockefeller Foundation (1997). Communications and Social Change: Forging Strategies for the 21st Century. A Report on a Rockefeller Foundation Bellagio Conference, April 21-25, 1997. Retrieved February 13, 2005 from http://www.rockfound.Org/Documents/250/cands2.html#methodology Rockwell, K. , Furgason, J., & Marx, D.B. (2000). Research and Evaluation Needs for Distance Education: A Delphi Study. Online Journal of Distance Learning Administration, 3(3). Retrieved July 27, 2003 from http://www.westga.edu/~distance/oidla/fall33/rockwell33.htinl Rogers, C.R. (1951). Client-centered counseling. Boston: Houghton-Mifflin. Rossman, G.B & Wilson, B.L. (1994). Numbers and words revisited: Being shamelessly eclectic. Quality and Quantity, 25(3), 315-327. Russell, D.R. (1993). Vygotsky, Dewey, and Externalism: Beyond the student/discipline dichotomy. JAC 75(1), 173-198. Retrieved July 14, 2005 from http : / / i ac. gsu.edu/j ac/13.1 / Art ic les/10 .htm Russell, M.P. (1999). Toward the Ideal Professional Master's Degree Program, Public Relations Review. 25(1). 101-111. Retrieved July 27, 2003 from ScienceDirect Database, U B C Library. Sadler-Smith, E. (2001). The relationship between learning style and cognitive style. Personality and Individual Differences, 30: 609-616 Sample, J A . (1984). Nominal Group Technique: An alternative to brainstorming. Journal of Extension, 22(2). Retrieved July 8, 2003 from http://www.ioe.org/ioe/1984march/iw2.html Saxe, G. B. (1991). Culture and cognitive development: Studies in mathematical understanding. Hillsdale, NJ: Lawrence Erlbaum Associates. Scardamalia, M . & Bereiter, C. (2002). Knowledge building. In Encyclopedia of Education (2nd Ed.). New York, N Y : Macmillan Reference, USA. Schafft, K A . & Greenwood, D.J. (n.d.) The Promises and Dilemmas of Participation: Action Research, Search Conference Methodology and Community Development. Ithica, N Y : Cornell University. Retrieved February 13, 2005 from http://www.cardi.cornell.edu/external/canal/Schafft_Greenwood.pdf Schifter, D., & Fosnot, C. T. (1993). Reconstructing mathematics education: Stories of teachers meeting the challenge of reform. New York: Teachers College Press. Scholl, H.J. (2004). Can SD models have greater relevance to practice when used within Participatory Action Research designs? 22nd International System Dynamics Conference (ISDC 2004), Oxford, U K July 2004. Retrieved February 13, 2005 194 from http://faculty.washington.edu/jscholl/papers/ISDC_2004_No_329.pdf Schreibman, V . (2001). Transforming of global capital: Breaking free of ideology in Cyberspace, Part II of II. Cyberspace Capital. January. Retrieved July 8, 2003 from http://sunsite.utk.edu/FINS/Cvberspace_Capital/CC-012.html Schwandt, T. R. (1990). Paths to inquiry in the social disciplines: Scientific, Constructivist, and Critical Theory Methodologies. In E.G. Guba (Ed.), The paradigm dialog (pp.258-276). Newbury Park, CA: Sage. Schwier, R. A . (1998). Schwiercourses, E D C M M 802, Unpublished manuscript, University of Saskatchewan at Saskatoon, Canada. Schwier, R. A . (2001). Catalysts, Emphases, and Elements of Virtual Learning Communities. Implication for Research. The Quarterly Review of Distance Education, 2(1), 5-18. Seels, B. (1989). The instructional design movement in educational technology. Educational Technology, 29(5), 11-15. ERIC Document Reproduction Service No. EJ 395 529. Siegenthaler, K. & Riley, K . (2002). What needs may be. Parks & Recreation. 37(12), p. 58-64 . Simmons, D. E. (2002). The forum report: E-learning adoption rates and barriers. In A. Rossett (Ed.), The ASTD e-learning handbook (pp. 19-23). New York: McGraw-Hil l . Simpson, T. L. (2001, November 8-10). Finding a home outside constructivist theory. Paper presented at the 43rd Biennial Convocation of Kappa Delta Pi, Orlando, Florida. Skinner, B.F. (1938). The behavior of organisms: An experimental analysis. New York: Appleton-Century-Crofts. Sloan, G. (1999). Good characteristics of a clinical supervisor: A community mental health nurse perspective. Journal of Advanced Nursing. 30(3). 713-723. Retrieved July 15, 2003 from EBSCO Database, UBC Library. Sloan-C View (2003). Coming to terms: A L N . 2(4). The Sloan Consortium. Retrieved October 28, 2004 from http://www.aln.org/publications/view/v2n4/coverv2n4.htm Smith, M . K . (2005). Carl Rogers: Core conditions and education. Infed. Retrieved July 14, 2005 from http://www.infed.org/thinkers/et-rogers.htm Smith, M . K . (2005). Fredrich Froebel. Infed. Retrieved July 15, 2005 from h ttp://www. infed. or g/thinkers/et-froeb. htm Smith, M . K . (2005b). Maria Montessori. Infed. Retrieved July 14, 2005 from http://www.infed.org/thinkers/et-mont.htm Smith, P.L., & Ragan, T. (1999). Instructional design. New York: John Wiley & Sons. Snelbecker, G. E. (1985). Learning theory, instruction theory, andpsychoeducational design. Lanham, M D : University Press of America, Inc. Solomon, D. J. (2001). Conducting web-based surveys: Practical assessment. Research & Evaluation, 7(19). Spiro, R. J., Feltovich, R. J., Jacobson, M . J., & Coulson, R. L. (1992). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. In T. M . Duffy & D. H. Jonassen (Eds.), Constructivism in education (pp. 57-75). Hillsdale, NJ: Erlbaum. 195 Stahl, G. (1999). Reflections on WebGuide. Seven issues for the next generation of collaborative knowledge-building environments. In C. Hoadley (Ed.), Proceedings of CSCL '99: The Third International Conference on Computer Support for Collaborative Learning (p. 600-610). Mahwah, NJ: Lawrence Erlbaum Associates. Steffe, L. P., & Gale, J. (Eds.). (1995). Constructivism in education. Hillsdale, NJ: Earlbaum. Steffe, L. P. & Kieren, T. (1994). Radical constructivism and mathematics education. Journal for Research in Mathematics Education, 25, 711-733 Suppe, F. (Ed.). (1977). The structure of scientific theories (2 n d ed). Urbana, IL: University of Illinois Press. Takala, ML, Hawk, D. & Rammos, Y . (2001). On the opening of society: Towards a more open and flexible educational system. Systems Research and Behavioural Science, 18, 291-306. Theory into Practice Database. (2005). Genetic epistemology (J. Piaget). Retrieved July 13, 2005 from http://tip.psychology.org/piaget.html Theory into Practice Database. (2005a). Constructivist theory (J. Bruner). Retrieved July 13, 2005 from http://tip.psychology.org/bruner.html Theory into Practice Database. (2005b). Social development theory (L. Vygotsky). Retrieved July 13, 2005 from httpV/tip.psychology.org/wgotsky.html Torrisi, G. and Davis, G. (2000). Online learning as a catalyst for reshaping practice: The experience of some academics developing online learning materials. The International Journal for Academic Development, 5(2), 166-176. Tubbs, N . (n.d.) Hegel and the philosophy of education. Retrieved July 13, 2005 from http:/Av-\v\v.vusst.hr/ENCYCLOPAEDIA/Hegel-PhiIosophv-of-Education.htm Turoff, M . , Hiltz, S.R., Bieber, M . , Fjermestad, J., & Rana, A. (1999). Collaborative Discourse Structures in Computer Mediated Group Communications. Journal of Computer Mediated Communication. 4(4). Retrieved July 28, 2003 from http://icmc,huii.ac.il/vol4/issue4/turoff.html University of Missouri Outreach and Extension. (2003). Nominal Group Techniques. Impact Leaflet No. 5b. Retrieved July 8, 2003 from http://outreach.missouri.edu/extcouncil/Impacts/5b.htm University of Western Australia (2001). Toward a definition of online learning at OWA. Academic Council Resoulution 39/01. Centre for the Advancement of Teaching and Learning. Retrieved October 28, 2004 from: http://www.catl.osds.uwa.edu.au/online/definition Vande Ven, A . H . & Delbecq, A . L . (1972). The nominal group as a research instrument for exploratory health studies. American Journal of Public Health. 69. 337-342. Vedros, K.R. (1979). The nominal group technique as a participatory, planning method in adult education. Ph.D. Dissertation, Florida State University. Vella, K., Goldfrad, C , Rowan, K. , Bion, J., & Black, N . (2000). Use of consensus development to establish national priorities in critical care. British Medical Journal. 320(7240), 976-981. Retrieved July 15, 2003 from EBSCO Database, U B C Library. 196 Vennix, J. A . M . , Gubbels, J. W., Post, D., & Poppen, H . J. (1988, July). A structure approach to knowledge acquisition in model development. Paper presented at the International Conference of the Systems Dynamics Society, La Jolla, CA. Vico, G. (1710/1988). On the most ancient wisdom of the Italians. (L.M. Palmer, Trans.). London: Cornell University Press. Vico, G. (1725/1968). The new science (T. G. Bergin, M . H . Fisch, Trans.). Ithaca, N Y : Cornell University Press. Virginia Institute of Government, (n.d.). CSA Training Materials: Nominal Group Technique (NGT). Retrieved July 8, 2003 from http://www.institute.virginia.edu/services/csa/nominal.htm Vygotsky, L. S. (1978). Mind in Society (Alexander R. Luria, Martin Lopez-Morillas, Michael Cole, James Wertsch, Trans.). Cambridge, M A : Harvard University Press. Walker, D. & Lambert, L. (1995). Learning and leading theory: A century in the making. In L. Lambert and D. Walker, D.P. Zimmerman, J.E. Cooper, M.D. Lambert, M.E. Gardner & P.J. Ford Stack, The Constructivist leader. New York, N Y : Teachers College Press, 1-27 Watzlawick, P. (Ed.). (1984). The invented reality. New York. N Y : W.W. Norton & Company, Inc. Wenger, E. & Snyder, W. (2000). Communities ofpractice: The organizational frontier. Harvard Business Review, (Jan.- Feb.).139-145. Wertsch, J. (1991). Voices of the mind: A sociocultural approach to mediation action. Cambridge, M A : Harvard University Press. Wikipedia. (2005). Constructivism (learning theory). Retrieved July 14, 2005 from http://en.wikipedia.org/wiki/Constructivism_%28pedagogical%29 Wikipedia. (2005a). John Dewey. Retrieved July 14, 2005 from http://en.wikipedia.org/wiki/John_Dewey Wikipedia. (2005b). Maria Montessori. Retrieved July 14, 2005 from http://en.wikipedia.org/wiki/Maria_Montessori Wiley, D. A . (2003). A brief history of the blog. In A . Kovalchick & K. Dawson (Eds.). Educational Technology: An Encyclopedia. Santa Barbara: ABCrCLIO. Retrieved August 3, 2003 from http://wiley.ed.usu.edu/docs/blog.pdf , Willis, J., & Wright, K .E . (2000). A general set of procedures for constructivist instructional design: The new R2D2 model. Educational Technology, 40(2), 5-20. White, K.W. & Weight B.H. (2000). The Online Teaching Guide. Needham Heights, M A : Allyn and Bacon. Wilson, B. G. (1997). The postmodern paradigm. In C. R. Dills & A . A . Romiszowski (Eds.), Instructional development paradigms. Englewood Cliffs NJ: Educational Technology Publications, in press (to be published in March 1997). Wilson, B. G. (1995). Maintaining the ties between learning theory and instructional design. Paper presented at the meeting of the American Educational Research Association, San Francisco, March, 1995. Retrieved February 20, 2005 from http://www.cudenver.edu/~bwil son Wilson, B. G., & Cole, P. (1991). A review of cognitive teaching models. Educational Technology Research and Development, 39(4), 47-64 Wilson, B. G., Osman-Jouchoux, R., & Teslow, J. (1995). The impact of constructivism 197 (and postmodernism) on ID fundamentals. In B. Seels (Ed.), Instructional design fundamentals (pp. 278). Englewood Cliffs NJ: Educational Technology Publications Winn, W. (1997). Advantages of a theory-building curriculum in instructional technology. Educational Technology, 37(1), 34-41. Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. J. Zimmerman and D. H . Schunk (Eds.), Self regulated learning and academic achievement: Theory, research, and practice (pp. 153-189). New York: Longman. Young, L .D. (2003). Bridging theory and practice: Developing guidelines to facilitate the design of computer-based learning environments, Canadian Journal of Learning and Technology, 29(3). 57-69. Retrieved September 10, 2004 from http://collection.nlc-bnc.ca/100/201/300/cdnjrn_learning_and _tech/2003/v29n03/www.cjlt.ca/conten t/vol29.3/ci It29-3_art4.html 198 Specific details on the full process will be supplied once you agree to participate. E-mail will notify all participant-experts before the start of each new round. Please note: approximate times are used in the description of this research. Rounds, depending on the group needs, may be slightly shortened or lengthened during the process. Round 1: Brainstorming (Duration: approx. 1 week) The goal of the first round is the identification by the participant-experts of the ideas or characteristics that apply to the three questions and the basic theme of the research. In this first round, it is important that the ideas are proposed independently and without discussion with other participant-experts. This round will be conducted by email between you, the participant, and me, the facilitator/researcher. Round 2: Presentation of the lists of brainstormed ideas and clarification regarding these ideas (approx. 1 week) In round 2 the combined list of ideas from all participant-experts will be presented to all participant-experts. The name of the participant will not be used. The purpose here is to guarantee anonymity, to enable a completely objective discussion of the ideas during this round. This round will be used to clarify any questions that you might have about ideas on the list. The goal here is to reach a common understanding of the content of the list by the entire group Round 3: Evaluation of the ideas (approx. 1 week) The purpose of this round is to give you the opportunity to evaluate and voice your opinions on the individual ideas listed so far. This will be accomplished through an online voting survey, with each item being given a scale of 0 to 5 (0 = little significance, 5 = important significance) with respect to the questions asked. There will be 3 such scales, one for each question in the research. Round 4: Internal publication and discussion of the results (approx. 1 week) In this round the authors of the ideas (via pseudonym) and the voting results will be open for viewing to all participant-experts. The results presented, in this round, can and are to be discussed, fully by the whole group. During this round, the intention is to have the group ratify the characteristics of constructivism, what constitutes instructional theory and the role of the instructor. A final vote will be carried out at the end of round 4. In addition, at the end of round 4, there will be a small online survey for you to fill our regarding your impressions of the NGT process and its online use. How much time will you need for participation? Of course, the time needed for you to participate can only be estimated. Overall, for the 4 rounds, you will need to set aside 1 -2 200 A P P E N D I X B D E L P H I A N D N G T D I A G R A M S Flowchart for the Delphi Method Start Vroblcm definition Select panel members based on the cxprriisc required Prepare and distribute questionnaireb-5 Analyze questionnaire responses Provide requested information and tabulated responses 0JDevelop final report 202 Flowchart for the Nominal Group Technique Start I Select Session Participants I Present and Discuss Tapic I Participants write respons.es I In Him, each panaipaiK provides a response wliich is tit'iej oil fiipc!iiiii Clarity ic>|H'«&.s acid tjinjiaalt' duplicates If o -Assign ionoi*ta number E U each I Kv*s{x*i dents cbosc iop issues & mnk by 'impLtii&i\i-£ I Aggregate all| rankings Wriic rcpttfi preuriik'S 203 APPENDIX C TYPES OF ASYNCHONOUS GROUP COMMUNICATION A G C can take many forms and below is a definition of different types of A G C . Distribution List: email is sent out to a group of people by one or several persons who have the list set up on their accounts. In this sense of the term it is not many-to-many communication, and therefore not strictly group communication. If people in the group have set up similar distribution lists and are aware of that fact, and there is some notion of the people in the distribution list constituting a group who can all contact each other as a group, then this does become a form of group communication. Mailing List: email is sent out to a group of people who have options to subscribe to the list (i.e. receive any mail sent to it), unsubscribe, or post messages that will be distributed to the whole list. Some people use the term mailing list to apply to distribution lists, but it is beneficial to maintain the distinction. Listserv: can either refer to the software of this name that allows a mailing list to function, or can be a generic name for all such software, or a generic name for all mailing lists. Conference: a particular subject of discussion is, by implication, part of a wider set of discussions being run using particular 'conferencing' software. The notion of a 'conference' implies that participant-experts will be able to: see the headers of all the contributions made on the particular subject, follow threads of debate, have some near-real-time opportunities for communication with other members of the conference (e.g. 204 'chat' features). If there are no near-real-time features then 'conference' is an up-market synonym for bulletin board. Bulletin Board: a bulletin board and a bulletin board system (BBS) have different meanings (See BBS below). A bulletin board is a specific area of a BBS usually dedicated to a specific topic, to which all messages sent by users are accessible to read. Discussion Area and Noticeboard are other names for a bulletin board. BBS: a BBS refers to some software on a particular machine, which allows users of other machines to dial in using modems and interface with the software. Normally this will be a very small number of users at one time (quite often there will only be one or perhaps two modems on the machine that is hosting the BBS). The software generally then presents the user with a menu of choices that may include some games, but will be likely to offer a set of discussion or software areas: the bulletin boards. Newsgroup: a Newsgroup is the name for a particular discussion area within the whole of USENET NEWS. Particular features of Newsgroups are the whole naming structure based on subject domains, the threading of messages, and the fact that the messages are moved around from server to server. This latter aspect of USENET means that for any individual their access to a Newsgroup is determined by whether their own network, or any network they have access to, has decided to carry the particular Newsgroup, or Newsgroups in general. Usenet can be accessed via Netscape or other WWW browser. Forum: a forum is a generic name for any topic or interest based electronic group communication whether by use of a mailing list or a bulletin board. Discussion areas on commercial areas, such as CompuServe, are referred to using this name, and fit the notion 205 o f bulletin board'. Forums have also been used by educators as a metaphor for deeper 'discussion' rather than the earlier idea of'posting' to a bulletin board. Interest Group: an Interest Group is another generic name for electronic group communication. Blog: A blog is a web page made up of usually short, frequently updated posts that are arranged chronologically, such as a 'what's new page' or a journal. The content and purposes of blogs varies greatly—from links and commentary about other web sites, to news about a company/person/idea, to diaries, photos, poetry, mini-essays, project updates, even fiction. Blog posts are like instant messages to the web. Many blogs are personal, "what's on my mind" type musings. Others are collaborative efforts based on a specific topic or area of mutual interest. Some blogs are for play. Some are for work. Some are both. Blogs are also excellent team/department/company/family communication tools. They help small groups communicate in a way that is simpler and easier to follow than email or discussion forums. Use a private blog on an intranet to allow team members to post related links, files, quotes, or commentary. Set up a family blog where relatives can share personal news. A blog can help keep everyone in the loop, promote cohesiveness and group culture, and provide an informal "voice" of a project or department to outsiders. 206 APPENDIX D E M A I L CORRESPONDENCE WITH DR. K . H O F F M A N N ...it is very interesting for us, that other researchers have found our project in the web and try to find out something more of it. So I will try to answer your questions, I hope that I can help you a little bit. Our survey had a case history: The first idea came from Prof. Schelle, who tried to formulate a project for a survey to the future of project management with international experts with the "Nominal Group Technique" in the late year 2000. Second was, that I have heard from it, when I was looking (for a friend) for a project that could be performed just with an interesting internet platform. So I found the article from Dowling, K . L . and Louis, R.D. St. : "Asynchronous implementation of the nominal group technique : is it effective?" In: Decision Support Systems, 29 (2000), pp. 229-248. Prof. Schelle and [I], we [met] together and founded a first part of the project in [the] German language with 10 experts on project management in 2001. A friend of mine realized [created] the web platform with small effort on php3 (script language) and mySQL (database). He had rich experience from programming a platform for a politic [political - democracy online] community (http://www.dol2day.de). In this first version of the survey we had six phases, in the whole we needed nearly 6 months to finish our survey (sorry, but all documents from this part are in German language). We had some problems, because we didn't know, whether the 207 single participants of our survey were looking on our websites - how often they did it, whether they had trouble with it and so on. So it needed a lot of mails and many phone calls, to bring most of the participants to fulfill, what they had to do in each of the six phases (for all participants it was voluntary work). The first results (in German language and only with 7 other experts, Prof. Schelle and me played the role of number 8 and number 9, three other experts had dropped out) were also very interesting, but this first version should be the "dress rehearsal" (???) for our international survey. We published the results in the autumn of 2001 (Hoffmann, K., Schelle, H. : Die Zukunft des Projektmanagements - Ergebnisse einer Expertenbefragung. In: Projektmanagement-aktuell 4/2001). We have learned in this first version of the survey, that • we need better tool support to see, how often and at what time each of the participants had made his/her last login on the project website, • we should use only four phases, to make the survey more efficient, • we should start each phase with a new mailing to all of the participants (altogether we needed more active guidance for our participants), • we should remember [to contact] some participants to fulfill their work, because the whole project is waiting for one or two responses. The[re] was no trouble with the principle of the Nominal Group Technique, because all theses were anonymous until the last phase of the survey; but it needed a lot of time to bring all work from all participants together. 208 On this experience and with an improvement of our administration tool, we started the second part of the project, an international survey in February 2002. [This refers to the report that is online at: http://wvvw.teamspring.net/S=7576adab219a6cb05ef46ff230ee9525/asynchron/]. A l l further steps you can read on the documentation, that I send to you with this mail. In addition to the technical documentation I send also the first mailing to our participants. If you have more questions or want to have more material (may be the text of our mails to the participants), please don't hesitate to contact me again. I hope, this can help in your research work. May be, my English is poor, but I hope, that you can understand the main aspects of our project. 209 APPENDIX E E M A I L CONTACT WITH STUDY PARTICIPANTS Invitation to Participate in Study Message From: Subject: To: Diane Janes Letter of invitation to join online research study Participant X 31 March 2004 22:26:09 March 31, 2004 Hello X : I am writing to invite your participation in an online study, designed to support my doctoral work exploring constructivism and online instructional design. The study, entitled Constructivism and Instructional Design: An Exploration Using an Asynchronous Online Nominal Group Technique invites you to participate online using the Nominal Group Technique (NGT). The purpose of this study is to identify what you believe are the tenets/presuppositions of constructivism as a theory of learning, and to suggest instructional approaches that flow from this theory for the development of online learning, including the role of the instructor. Using a consensus-building methodology, the online NGT, this process would require visiting the research website four times over a four to five week period beginning in April 2004. The total time commitment would be approximately 4-5 hours throughout the project. I hope you will give serious contemplation to participating—I look forward to your contribution in attempting to reach a consensus with colleagues, on what constructivism means and its implications for the design of online learning programs and the role of the instructor. Thank you for your consideration. I have included the informed consent form with greater detail about the project in the body of this e-mail message. If you decide to participate, please type your name where indicated, and return the form to me via e-mail by April 7, 2004. It is anticipated the first round will begin by April 15th but that will be confirmed by email to you, along with the web URL and your login information. If you have any questions for me, before joining the study, please don't hesitate to get in touch by email or telephone. Cheers, Diane Diane P. Janes, M . Ed, PhD (candidate) Educational Studies, Faculty of Education University of British Columbia Assistant Professor Centre for Distributed Learning (CDL) Extension Division, University of Saskatchewan Room 413, Williams Building 221 Cumberland Ave. N . 210 frame. Also you enter the Round 2-discussion area, you will either see a link to Round 2A or Round 2B. From here on in the study will be conducted in 2 smaller groups (19 people each) concurrently until the final Round 4 when we will all meet in the one discussion area. But you will only see the work and votes of your own group until the end Round. As 1 mentioned in my first email to you the NGT is a very structured process, so as facilitator 1 will continue to keep people on track (by email and posting) and on the research timeline. My thanks, again, for keeping to the structure that NGT requires. In Round 1 you were asked to brainstorm your thoughts on three questions. These brainstorms produced a list of points (over 400 actually), some of which were merged (if they were deemed similar by the facilitator). The remaining points appear on your list for Round 2. Where necessary and possible, the points have been grouped under emerging themes and sub-themes. When you enter WebCT and the your discussion area, you will find the list generated by Round 1 under links (Round 2A or Round 2B) on the left navigation bar. This will be a pop up window broken down by the main themes, and printable by your browser. In Round 2 your task is to seek clarification, through a group discussion, on any of the items on the list that you find to be vague or unclear. This is also an opportunity for you to suggest points that should be merged, deleted or added to the list (new points). Please avoid commenting on your own postings unless asked for clarification by another participant. You may, during this round: • post a question of clarification to another participant, • post a recommended merger of points (using the numbers assigned to a points as reference)*, • suggest the deletion of a point (again referring to the point's assigned number)* or • add a point that you feel is missing from the list. • Suggestions for merger or deletion will not be carried out if another other group member disagrees with the suggested merger/deletion. Additions to the list will be automatically included. Note: This Round may require you to check back to the online discussion area several times over the 2 week period. Round 2 is designed to be an active discussion on the ideas generated in Round 1. As with Round 1, to enter Round 2 use the following U R L : http://det.cstudies.ubc.ca/DJT/index.html Follow the link to Round 2. You will be asked for a login and password as you enter the group discussion area. Your login ID/password for the research study will remain the same as in Round 1. Please contact me by email if you have any questions or need clarification. On the weekend of May 8-9th (MST), the Round 2 discussions will be gathered, and you will be asked to vote on the final list using an online survey tool (Round 3). Details on this next round, including instructions on using the online survey tool will be sent to you, by email, at the beginning of Round 3. Thank you for agreeing to continue into Round 2 and I look forward to your continued excellent comments and ideas. Cheers, Diane Diane P. Janes Assistant Professor Centre for Distributed Learning (CDL) 214 Round 4 - Introductory Message Message From: Subject: Diane Janes Round 4 begins... Participant X To: 18 July 2004 22:53:13 H i X , Many thanks for your participation in Round 3. The data has finally been tabulated and is ready for the final formal round of the online Nominal Group Technique, Round 4. In this round the voting results from Round 3 will be open for viewing by all participant-experts. You will find 3 threads in the Discussion Forum entitled 'Round 4'. You are asked to comment on the results, to raise questions about any surprises/unexpected outcomes, and to explain why you agree or disagree with the results. You will also be asked in this Round to define your own personal 'level of consensus' for the study, that is, what is the lowest % you feel confirms a consensus on a statement and, finally, to affirm (or not) your agreement and/or consensus with the voting results of Round 3. A copy of the Round 3 Results can be found on the left navigation bar of the WebCT main page. A copy (in MSWord) is also attached to a posting directly inside of the Round 4 discussion area. This Round is open to any and all of your comments on the vote and its results. For this Round we go back to the Thesis Website. As with Round 1 and 2, to enter Round 4 use the following URL: Follow the link to Round 4. You will be asked for a login and password as you enter the group discussion area. Your login ID/password for the research study will remain the same as in previous rounds: Login: estudyX Password: XonlineX Please contact me by email if you have any questions or need clarification. Thank you for agreeing to continue the final phase of this study. I look forward to your continued excellent comments and ideas. Cheers, Diane PS. Just a reminder, the only remaining activity, after Round 4, will be a short survey to evaluate your experiences within the online NGT process. This should take about 10 minutes to complete and an invitation will be sent to you to take the survey, via email, toward the end of Round 4. 1 will be using the same e-survey tool as in Round 3. Completing this survey will be very helpful as it will allow future researchers to build on this process by learning from our experiences. Diane P. Janes Assistant Professor Centre for Distributed Learning (CDL) Extension Division, University of Saskatchewan Room 413, Williams Building http://det.cstudies.ubc.ca/DJT/index.html 222 APPENDIX F E M A I L WITH SUPERVISOR RE: ROUND 2 REVISION Message From: Subject: To: Tom Sork <tom.sork@ubc.ca> RE: RE: Thesis question-Diane Janes 23 April 2004 16:02:32 Hi, Diane: Your summary reflects what I remember from our conversation. 1 think this will work and be highly defensible. Have a good weekend. Tom. Original Message From: Diane Janes [mailto:Diane.Janes@extfc.usask.ca] Sent: April 22, 2004 5:46 PM To: Tom Sork Subject: Re: RE: Thesis question... Hi, Tom...thanks for taking the phone call just now...here is what 1 am thinking... - Round 1 was 38 participants... - Round 2 will begin the creation of 2 groups (2A & 2B) which will continue for the remainder of the study - Each group 2A and 2B get their own generated list for Round 2 to ask for clarification (merging, deletion or addition) - Each group 2A and 2B will vote on their own modified list using my online survey tool... Based on ranking and other criteria, I will choose the top x items from each vote, and have them combined on to one list (delete any overlap) This final list is presented to the full group for consensus and discussion - could end up that another vote on the new list might be necessary but will let the group decide that at the time...'emergent design' Final list is agreed to and study is complete...;-) Article that split groups and used Modified NGT: Nelson, J. S., Jayanthi, M . , Brittain, C. S., Epstein, M . H. , Bursuck, W. D. (2002). Using the Nominal Group Technique for homework communication decisions. Remedial & Special Education. 25(6), p. 379-387. Well, all for now...I'm off to keep creating the list, and then I'll sort out the rewriting of the thesis chapter I've got underway as I am data collecting...;-) Great to talk to you...cheers, Diane Tom Sork <tom.sork@ubc.ca> writes: >Hi, Diane...That is a bit complicated. I wonder what dividing the folks into two groups does to the >integrity of the NGT? If the purpose is to reach consensus, then how can that be done when the group >is broken down into two? 225 >The important thing is that you are able to defend whatever choice you make. There is certainly a good >practical reason for subdividing, but you need to consider carefully the methodological implications of >doing that. If you see a way of arriving at consensus (across the two groups) by the end of the process, >then I'd say "fine...go ahead." > >I think it might be somewhat problematic to treat these as two separate groups from now on. You might >end up with two different consensuses! And I wouldn't know what to say about the two...you can >observe how similar or different they are...or concentrate your analysis on only the elements of >agreement...so that might be a procedural solution. > >I wonder if there was any example you found in the literature where NGT was completed using more than >a single group. > >Sorry for all the questions and speculations, but I don't have a ready answer for you. The key is to be able >to justify—methodologically—whatever you decide to do, so think carefully of the implications for the >integrity of your study. > >Tom. > > Original Message >From: Diane Janes [mailto:Diane.Janes@extfc.usask.ca] >Sent: April 22, 2004 4:33 PM >To: tom.sork@ubc.ca >Subject: Thesis question... importance: High > >Hi, Tom...I am currently synthesising the data for Round 1-18 pages of it and that is only question >1 ...yikes! ;-) > >But I did have a question for you...given the numbers I am working with, I think putting them into one >discussion forum would be too much...so I want to take this opportunity to divide the group and put them >into 2 separate discussion areas for Round 2...I will divide the data compilation into 2 and they will see >one or the other depending which group they ended up in, which would allow each group about 19 people >to talk about the ideas submitted from only those 19 in that group... > >This would give me an opportunity to see if each group comes to the same/similar concensus... > >What do you think? I am trying to get this done for tonight to start the 2nd round tomorrow... > >Your thoughts would be appreicated...cheers, Diane Message From: Diane Janes Subject: update thesis data collection... To: Tom Sork 14 May 2004 13:41:10 Hi , Tom...I hope all is well with you... Just an update...Round 2.2 is underway as of this morning...I have broken the group down into 11 smaller groups - each with an element/category or part of an element/catagory if the item list was greater than 30-35 items...I should start getting responses over the weekend, so I am looking forward to this next phase...I am, I estimate about 1/2 way thru the process, although Rounds 3 and 4 should be briefer and tighter...;-) 226 APPENDIX G S U R V E Y INSTRUMENT 228 Page 1 of 21 </b>Welcome to Round 3. The questions below, and on the subsequent pages, are a synthesis of the statements generated in Rounds 1, 2 and 2.2 over the past 8 weeks. You are being asked below to do 3 things... </b> <p>After reading each statement: <p>l. <i>Agree</i> or <i>Disagree</i> that this statement is an essential core or primary feature of constructivism and online learning [C&OLL] <p>2. Rank each statement on its importance as a primary feature of constructivism and online learning [C&OLL] - with <i> 1 - essential, 2 - very important, 3 - important, 4 - somewhat important, 5 - not important</i>. <P>3. Indicate whether you consider these views you have expressed, regarding the statement above, are more your own <i>personal view</i> (a minority view) or more of a <i>global view</i> (a majority view) of constructivism and online learning [C&OLL] <P>PLEASE NOTE: you are being asked to click on ALL of the 3 comments with respect to each statement. The software <u>will not allow you to proceed</u> to the next page until all buttons for each statement have been clicked. Check your page before clicking on NEXT; <p>Page 1 of 4 Primary feature of C&OLL <b>Constructivism and Knowledge</b> Agree Disagree Knowledge does not exist ,-, 0 independent of the learner. v ' v ' Rank importance to C&OLL 1- 2-very 3- 4-somewhat 5-not essential important important important important Would you consider your comments on the above statement, a personal view (held by yourself - a minority view) or a global view (a commonly held belief about constructivism and online learning - a majority view)? Q personal view O global view Primary ' feature Agree Disagree Learners are encouraged to seek knowledge and experience from different (...) O types of sources and .^ perspectives. Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important O O O O' o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary _ . . . c , . ' Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important The process is iterative and lifelong so that our own p. r ^ r ^ p. - p "meaning-making" is K J - ' constantly changing. • </b>Would you consider your comments a personal view or a global view?</b> C) personal view 229 Page 2 of 21 O global view Primary feature Agree Disagree Learners interact with a media-rich and resource-rich environment in building knowledge. Rank importance 1- 2-very 3- . 4-somewhat 5-not essential important important important important </t»Would you consider your comments a personal view or a global view?</b> O personal view O global view Fosters active/higher-level thinking and reflection. Primary feature Agree Disagree o o Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Encourages/values the development of personal understanding/knowledge construction by the learner. Primary feature Agree Disagree Rank importance 4-1- 2-very 3- somewhat 5-not essential important important important important o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important 230 Page 3 of 21 Understanding that there are multiple ways that 'knowledge' is absorbed and constructed. 0 o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Builds on prior learning (knowledge) of individuals. Primary feature Agree Disagree o o Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view P r i m a r y Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important, important important important Critically merges the learner's knowledge of the world with disciplinary insight and experience. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary „ . . . f e a t u r e Rank importance 4-<b>Constructivism and Active , ^ _, , , „ Participation</b> . n . 1 \ . , . 2 ^ , . 3 " , somewhat 5-not Agree Disagree essential important important important important Emphasise/facilitate active participation of learners with the content (dialogue) and with each other (interaction); with the intent () O O O O O O of helping build a high degree of ' ' relevance to individuals and the group. 231 Page 4 of21 </b>Would you consider your comments-'a personal view or a global view?</b> O personal view O global view Primary feature of C&OLL Rank importance to C&OLL <b>Constructlylsm and ^ 2 . v e r y 3_ SOmewhat 5-not Authentic Learning</b> A g r e e D i s a g r e e essential important important important important Emphasises experiential, authentic learning, which i s f) © © © © © O decided on by learners or negotiated with them. </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view ' ^ r i m a r y Rank importance feature . 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Open-ended discussions based on critical inquiry/authentic intellectual work. © © o © © </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Primary . . . , , ' Rank importance feature K 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Builds skills and abilities while working on authentic © © O O © O O tasks and problems. </b>Would you consider your comments a personal view or a global view?</b> Q personal view ©global view P r i m a r y Rank importance feature 232 Page 5 of 21 Learning activities are "authentic" and require reflection. Agree Disagree o o essential o. 2-very 3- 4-somewhat 5-not important important important important o o </b>Would you consi © personal view O global view der your comments a personal view or a global view?</b> Primary feature Agree Disagree Provides a safe environment, encouraging risk-taking and authentic dialogue. Rank importance 1- 2-very 3- 4-somewhat . 5-not essential important important important important o </b>Would you consider.your comments a personal view or a global view?</b> O personal view O global view Primary feature Agree Disagree Learners should have opportunities to solve complex, realistic problems similar to those that the learners themselves would encounter outside the classroom. o Rank importance 1- 2-very 3- somewhat 5-not essential important important important important o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view 1 Primary . . . , . ' Rank importance feature <b>Constructivism and Multiple ^ 2 _ v e r y 3_ s o m e w h a t " 5 _ n o t erspec ives / ^ g r e e Disagree essential important important important important Sharing multiple perspectives and hearing the constructions of Q Q Q Q Q O O others allows one to question our original assumptions and to https://www.hostedware.corn/secure/hs/Results/9779.html 2005-09-29 233 Page 6 of 21 develop a shared meaning. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary . feature Rank importance 4-1- ' 2-very 3- somewhat 5-not Agree Disagree essential important important important important Learning is constructed as a result of the process(es) where multiple perspectives are • © © © © © © © revealed, discussed, then reconstructed. </b>Would you consider your comments a personal view or a global view?</b> O personal view © global view Round 3: Constructivism and Instructional Design (continued) <p>Page 2 of 4 Primary feature Rank importance Encourages exploration of multiple paths and forms and facilitates opportunities for varying experiences and pathways through the subject matter. Agree Disagree © © 1- 2-very 3- somewhat 5-not essential important important important important © © © © </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Primary Rank importance feature 4-1- 2-very 3- somewhat 5-not Agree Disagree essential important important important important Constructivism assumes a subjectivist view of the world, https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 234 Page 7 of 21 even a post-modern one of multiple realities and no "grand O O O narratives." </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary n . . . , . ' Rank importance feature v <b>Constructivism and Self- 1- 2-very 3- 4-somewhat 5-not Directed Learning</b> Agree Disagree essential important important important important Provides and balances self-directed and collaborative O O O O O O O learning opportunities. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view P r , m a r Y Rank importance feature <b>Constructivism and ^ 2 . v e r y 3 . s o m e w h a t 5-not Collaboration</b> A g r e e D j s a g r e e essential important important important important Social negotiation of meaning can happen through p 0 p p r - , asynchronous / synchronous v ' ' J " v collaborative tools. </b>Would you consider your comments a personal view or a global view?</b> O personal view (3 global view P r i m a r y Rank importance feature 1- 2-very ' 3- 4-somewhat 5-not Agree Disagree essential important important important important Emphasizes co-operative learning (collaborative ^ , construction of common ' w K - ^ projects or representations). </b>Would you consider your comments a personal view or a global view?</b> O personal view 235 Page 8 of21 O global view Primary feature Agree Disagree Support the goals of learners by facilitating appropriate forms of learner collaboration and discussion. O Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important o </b>Would you consider your comments a personal view or .a global view?</b> O personal view O global view <b>Constructivism and Community</b> Knowing is mediated by community/culture and the tools of that community and culture, so be aware of and exploit cultural differences. Primary feature Agree Disagree o o Rank importance 1- 2-very 3- somewhat 5-not essential important important important important </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Seeks engagement with a community of learners or practice. Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important r" </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature <b>Constructivism and the https://www.hostedware.com/secure/hs/Results/9779.html Rank importance 2005-09-29 236 Page 9 of 21 Instructor</b> Steps aside and allow the class to figure out how it will go down the path that is established. Agree Disagree © 1- 2-very 3- somewhat 5-not essential important important important important 0 O O o </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Primary feature Agree Disagree-Provides, opportunities, p but not always answers. Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Supports multiple conceptions and promote critical thinking that stretches boundaries. Primary feature Agree Disagree Rank importance' 1- 2-very 3- 4-somewhat 5-not essential important important important important n O </b>Would you consider your comments a personal view or a global view?</b> © personal view ©global view Flexible in responding to different starting points for different individual learners. Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important </b>Would you consider your comments a personal view or a global view?</b> © personal view https://www.hbstedware.com/secure/hs/Results/9779.html 2005-09-29 237 Page 10 of 21 Oglobal view Primary _ . . . feature R a n k l m P ° r t a n c e 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Practised in the art of evidence-based teaching O O O O O (...) O and instructional design. </b>Would you consider your comments a personal view or a global view?</b> O personal view Oglobal view Primary . . . , . ' Rank importance feature r l 1 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Uses a learner-centred approach to teaching, which r . , r - , r , r-. involves sharing authority ' ' J v ' "y with learners. </b>Would you consider your comments a personal view or a global view?</b> O personal view Oglobal view P r i ™ a r y Rank importance feature v 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Becomes a metacognitive coach for O O O O O O O learners. </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view P / i m a r y Rank importance feature 4-1- 2-very 3- somewhat 5-not Agree Disagree essential important important important important Works to nudge students 2005-09-29 https://www.hostedware.com/secure/hs/Results/9779.html 238 Page 11 of 21 toward the kind of intellectual responsibility they need to r % become self-motivated, self- • y K J J actualised learners. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary n . . . „ . ' Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Offers opportunities for students to follow their O O O O O O O own interests. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Round 3: Constructivism and Instructional Design (continued) <p>Page 3 of 4 l ! r i m a r y Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Encourages the voices of all learners and is respectful of what the learner can O O O O O O O contribute to the learning enterprise. </b>Would you consider your comments a personal view or a global view?</b> Q personal view O global view The instructor is flexible enough to support student expressions of different levels of knowledge. Primary feature Agree Disagree o Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important o https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 239 Page 12 of21 </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature <b>Constructivism and Instructional Design</b> Agree Disagree Provide activities at the right level of challenge for O ' (1) learners. Rank importance 1-. 2-very 3- 4-somewhat 5-not essential important important important important O </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Provide a variety of ways of introducing course material so those students with different learning styles are accommodated. Primary feature Agree Disagree o Rank importance 4-1- 2-very 3- somewhat 5-not essential important important important important O o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature Agree Disagree Should accommodate a shift in expected learning O O outcomes </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important o c n Primary ' feature Rank importance https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 240 Page 13 of 21-1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Should create an environment with a social O, O O O O O O context. </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Primary „ . . . c . Rank importance feature H 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential' important important important , important Provides opportunities for dialogue between novices O O O O O O ( ) and experts. </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Pn™arY Rank importance feature K 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Provides a clear description of the rules of the course so that r- , r , ^ ( - v . all learners understand the v"' v ' ^ O playing field. </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view P r i ™ a r y Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important The core of the design ^ , r-. rs, r-, is learner-centred. </b>Would you consider your comments a personal view or a global view?</b> O personal view Oglobal view 2005-09-29 https://www.hostedware.com/secure/hs/Results/9779.html 241 Page 14 of 21 feaTure Rank importance <b>Constructivism and 1- 2-very 3- 4-somewhat 5-not Assessment</b> Agree Disagree essential important important important important Assessments should accommodate the needs of O O O O O O O diverse learners. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary _ , . . f e a t u r e Rank importance 4-1- 2-very 3- somewhat 5-not Agree Disagree essential important important important important Assessment should avoid overly narrow measurement of discrete skills or knowledge O O O O O O O . without a more complete context of interpretation. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view ^ n m a r y Rank importance feature K 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Assessment systems should examine a learner's full range of knowledge, skill, and O O O O O O O attitudes toward a domain of expertise. </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary R a n k j m p o r tance feature 4-1- 2-very 3- somewhat 5-not Agree Disagree essential important important important important https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 242 Page 15 of 21 Learners should have access to assessment criteria and methods, and be able to negotiate at least some of the terms of the assessment. ( J O </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature Agree Disagree Assessment should often include learners' reflections on p p their goals and accomplishments. Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important o </b>Would you consider your comments a personal view or a global view?</b> C) personal view O global view Learners may take away more from the experience than originally anticipated. Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important O O </b>Would you consider your comments a personal view or a global view?</b> . O personal view O global view Rank importance </b>A variety of learner performances and products (both individual and group-based) may be suitable for assessment, including: </b> Primary feature <b>Constructivism and 1- 2-very 3- 4-somewhat 5-not Assessment</b> Agree Disagree essential important important important important Live performances O O O O O O O </b>Would you consider your comments a personal view or a global view?</b> O personal view https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 243 Page 16 of 21 Ogloba l view Primary feature Agree Disagree Rank importance 1-essential 2-very important important Creative works 4-somewhat important o 5-not important </b>Would you consider your comments a personal view or a global yiew?</b> O personal view Ogloba l view Round 3: Constructivism and Instructional Design (continued) <p>Page 4 of 4 Primary feature Agree Disagree Rank importance essential 2-very important important Inquiry reports 4-somewhat important 5-not important </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Primary feature Agree Disagree Problem-based p Q learning Rank importance 1- 2-very 3-essential important important 0 O 4-somewhat important © 5-not important o </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Primary feature Agree Disagree Performance assessments Q Q and skill demonstrations Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important Important important © © © © https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 244 Page 17 of 21 </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Primary . . . , . ' Rank importance feature K 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important • important important Portfolios/e-portfolios or collections of work © © © © © © O samples </b>Would you consider your comments a personal view or a global view?</b> O personal view Oglobal view Primary „ . . . feature . R a n k i m P o r t a n c e 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important S m i n T 6 " ° ® © © © © © </b>Would you consider your comments a personal view or a global view?</b> O personal view Ogloba l view Primary n , . , . ' , . Rank importance feature K 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Guided-discovery ~ ^ m m m p 0 learning ^ ^ ±J W <J -,J </b>Would you consider your comments a personal view or a global view?</b> O personal view Oglobal view Primary _ , . . , ' Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Role-playing p r . p 0 Q r , p (gaming) . ° ° ° ° U ~ y~' ' https://www.hostedware.com/secure/hs/Results/9779.html 2005-09^29 245 Page 18 of 21 </t»Would you consider your comments a personal view or a global view?</b> O personal view O global view P r i m a r V Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important. important , important S s 9 o o o o o o o </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important Small group inquiry exercises, reported out to the whole class </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Primary Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Research papers and presentation of findings O O r </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view P r i r n a r y Rank importance feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important Independent reading https://wvvw.hostedvvare.com/secure/hs/Results/9779.html . 2005-09-29 246 Page 19 of21 framed as self-guided research O o </b>Would you consider your comments a personal view or a global view?</b> O personal view Q global view Asynchronous discussion Primary feature Agree Disagree Rank importance essential 2-very 3-important important © © 4-somewhat important © 5-not important </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Peer teaching Primary feature Agree Disagree O © essential © 2-very important © Rank importance important © 4-somewhat important © 5-not important </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Primary feature Agree Disagree Simulations © © essential © 2-very important © Rank importance important © 4-somewhat important ® 5-not important O </b>Would you consider your comments a personal view or a global view?</b> ©personal view ©global view • Primary feature Agree Disagree https://www.hostedware.com/secure/hs/Results/9779.html Rank importance 1- 2-very 3- 4-somewhat essential important important important 5-not important 2005-09-29 '247 Page 20 of 21 Evidence-based practice G O © © © © </b>Would you consider your comments a personal view or a global view?</b> © personal view ©global view Primary feature Agree Disagree Webquests © O essential © 2-very important © Rank importance important © 4-somewhat important © 5-not important O </b>Would you consider your comments a personal view or a global view?</b> O personal view O global view Synchronous on-line sessions Primary feature Agree Disagree Rank importance 1- 2-very 3- 4-somewhat 5-not essential important important important important r </b>Would you consider your comments a personal view or a global view?</b> © personal view ©global view .! Content/topic development/resource gathering Rank importance Primary feature 1- 2-very 3- 4-somewhat 5-not Agree Disagree essential important important important important </b>Would you consider your comments a personal view or a global view?</b> O personal view ©global view Do you have any final comments or thoughts on the items that you just rated? https://www.hostedware.com/secure/hs/Results/9779.html 2005-09-29 248 https://vvwvv.hostedware.com/secure/hs/Results/10329.html Demographics <p>(Please click on the button that most represents your answer.) Gender? C Male f Female Highest level of education attained? f Masters Masters working toward Doctorate Doctorate f Other If other, please indicate the qualification last attained. Describe the type of work you are currently engaged in. Overall Impression of the Nominal Group Technique (NGT) What did you like MOST about the online NGT? And why? What did you like LEAST about the online NGT? And why? What would you have changed? And why? 1 of 8 249 9/29/05 3:08 P M https://vvww.hostedvvare.com/secure/hs/Results/10329.html Did you find it difficult to keep engaged in the NGT process? f yes C sometimes f no If yes or sometimes, what kept you from full engagement? (please note as many reasons as possible) If no, what kept you fully engaged? (please note as many reasons as possible) Usefulness of WebCT Using a scale of 1 to 5, with 1 being 'not at all well' and 5 being 'excellent', how well did you think WebCT worked as a research-gathering tool? not at all well f somewhat well ? well C very well ? excellent Why did you you rate the usefulness of WebCT as you did? (please note all of your reasons) 2 of 8 250 9/29/05 3:08 P M https://www.hostedvvare.eom/secure/hs/Results/l 0329.html How well would you rate the ease with which you were able to access the WebCT discussion board? ( l=not at all well; 5 = excellent) C not at all well f somewhat well . C well f very well <~ excellent Why did you rate your ease of access as you did? (please note all reasons) What advice would you give to future researchers using WebCT as a research-gathering tool? Did you experience any difficulty when using the WebCT tool? f yes no If yes, what was the problem and how was it addressed? (please explain) Overall Impressions of the Study How would you rate the instructions that were given to you about the process, on the NGT website? hot at all clear ^ somewhat clear clear f very clear f excellent 3 of 8 251 9/29/05 3:08 P M https://vvvv\v. hostedvvare.com/secure/hs/Results/10329.html Why did you choose this rating? How would you rate the email instructions that were given to you about the process? f not at all clear f somewhat clear f clear f very clear *~ excellent Why did you choose this rating? I How would you rate the 'gentle reminders' that were sent to you during the process? *~ not at all useful <"* somewhat useful useful C very useful f excellent Why did you choose this rating? The NGT process uses a consensus building as a basic method of bringing forth questions and encouraging dialogue. How would you rate this methodology for generating/encouraging discussion? f not at all useful somewhat useful useful 4 of 8 252 9/29/05 3:08 PM https://wvvvv.hostedware.com/secure/hs/Results/10329.html f very useful excellent Why did you rate this methodology in this way? (please give all reasons) I How well did you feel the study addressed the issues that are salient for constructivism and online learning? How well did the NGT meet your expectations? f not at all well ^ somewhat well f well very well excellent Why did you choose this rating? The Rounds How useful did you find Round 1? f not at all useful f somewhat useful useful very useful f excellent 5 of 8 253 9/29/05 3:08 P M https://\vvvvv.hostedware.com/secure/hs/Results/ 10329.html Why? How useful did you find Round 2? <* not at all useful ^ somewhat useful C useful very useful C excellent Why? How useful did you find Round 2.2? f not at all useful somewhat useful C useful ^ very useful excellent Why? How useful did you find Round 3? C not at all useful f somewhat useful ? useful . very useful <* excellent 6 of 8 254 9/29/05 3:08 PM https://w\v\v. hostedware.com/secure/hs/Results/ 10329.html Why? How useful did you find Round 4? not at all useful C somewhat useful useful very useful *~ excellent Why? Usefulness of Online Survey Tool Using a scale of 1 to 5, with 1 being 'not at all well' and 5 being 'excellent', how well did you think this online survey worked as a research-gathering tool? f not at all well <"* somewhat well well C very well excellent Why did you choose this rating? Did you experience any difficulty when using the online survey tool? f yes 7 of 8 255 9/29/05 3:08 PM https://\v\v\v.hostedware.com/secure/hs/Results/ 10329.html f no If yes, what was the problem and how was it addressed? (please explain) Final Questions In total how much time did you commit to the NGT process? ^ less than 1/2 hour f between 1/2 and 1 hour f more than 1 hour but less than 2 hours <** more than 2 hours but less than 3 hours more than 3 hours but less than 4 hours <"* more than 4 hours less than 5 hours more than 5 hours If more than 5 hours, please estimate the total time you spent on the NGT. Please add any final comments on any aspect of the process or the study that you may have. 8 of 8 256 9/29/05 3:08 P M A P P E N D I X H I D E A S T A T E M E N T S G E N E R A T E D I N R O U N D 1 Sub-Themes - Instructional Design Design Context 1. Authentic tasks/problems (estudy29) (estudy20) 2. Communities of practice (estudy29) 3. Learning as social and dialogical process (estudy29) 4. Opportunities for exchange between experts and apprentices (estudy29) 5. Knowledge community (estudy29) 6. learner rather than teacher centred (estudy07) 7. involvement in meaningful contexts is critical for student understanding (estudy20) 8. multiple perspectives (estudy31) 9. construction, maintenance and facilitation of a true SOCIAL C O N T E X T is perhaps the biggest challenge in online instruction, especially when the learners are culturally diverse (estudy 18) Design Tasks 10. Information gaps stimulate learners to explore, analyse and synthesise (estudy08) 11. Must address learners attitude and perception towards learning (estudy08) 12. requires feedback loops regarding the learning process which likely includes some metacognition (estudy 10) (estudy06) 13. Collaborative tasks where students work together, share perspectives, and contribute to solutions (estudy06) (estudy23) (estudy08) 14. Discussion to have some mechanism for facilitation and some "rules of engagement" (ideas regarding process, netiquette, etc.) (estudy 10) 15. Learning supports provided so learner's motivation is maintained and time is somewhat managed (estudy03) 16. design media to support dual coding (estudy08) 17. visual representation of ideas e.g. collaborative concept map (estudy08) 18. arrangement of tasks from simple to complex in line with Bloom's taxonomy (estudy08) 19. use granularity in tasks (estudy08) 20. use cognitive task analysis (estudy08) Pedagogy 21. scaffolding of learning onto prior experiences(estudy22) 22. Scaffold defined processes of inquiry, knowledge construction (estudy06) 23. Challenges to current thinking, followed by opportunities to build alternative understandings (estudy06) (estudy23) 24. As learners become more self-directed, the objectives may be negotiated in some areas or may not be achieved in the same order as one might have planned as a designer (estudy03) Activities 25. feature different types of activities to accommodate a range of learning styles (estudy07) 26. requires reflection on what has been learned/personal goals/needs (estudy 10) (estudy06) 27. meaningful problem-solving activities (estudy06) 28. Finding activities that are designed at the right level of challenge, will stimulate curiosity and motivation (estudy 14) 257 29. Individual tasks with opportunities to see how others solved them, followed by discussion and reflection (estudy06) (estudy08) 30. Guided activities where the content is shown to be useful in addressing problems and tasks (estudy06) Sub-Themes Social/Community/Culture Social Aspects 1. learning is a product of negotiation with others (estudy25) (estudy03) (estudy20) (estudy34) (estudy04) (estudy21) (estudy05) (estudy02) 2. meaning is constructed by engagement and interaction with the world/issues/context (estudy06) (estudy03) (estudy20) (estudy34) (estudy04) (estudy20) 3. meaning is constructed with other people (estudy06) (estudy03) (estudy20) (estudy34) (estudy04) 4. Socially mediated process (estudy22) (estudy23) 5. Interaction, collaboration, challenge & feedback from others is key to helping students to establish accuracy & application of their ideas (estudy05) (estudy34) (estudy02) (estudy23) (estudy04) (estudy 14) (estudy06) 6. social aspects of learning can shape the nature and quality of learning (estudy 18) (estudy23) (estudy04) 7. our existing understandings or mental constructs (i.e. prior knowledge) may be challenged by the mental constructs of others (estudy02) (estudy23) 8. Sharing multiple perspectives and hearing the constructions of others allows one to question our original assumptions (estudy23) 9. people *can* learn without direct cultural/social mediation (e.g., independent study, not that they can escape their prior learning and influences of culture) but that we all learn more easily in social contexts where we can seek assistance (the idea of ZPD) and verify our thinking with others (estudy26) Community and Culture 10. the processing of knowing is mediated by community and culture and the tools of that community and culture (especially language) (estudy26) 11. highly cultural - we construct meanings in very different ways based upon belief systems, societal and community group values. Even behaviourist systems of education are structured for the cultural in which they are situated, and the experience one has in such systems impact the construction of knowledge (estudy 16) 12. a safe environment that allows for diversity and inclusively otherwise one does not construct their own meanings - they may their negate their own worldview in favour of the dominant one (estudyl6) 13. knowledge is constructed by individuals actively in order to give meaning to socially accepted and shared notions and that an individual's construction of knowledge is tentative (estudy26) Group Work/Collaboration 14. Placing students in groups and encouraging students to 'own what they know' has become much more typically used as an educational tool in the constructivist era (estudyl5) 15. many put students into groups because they have less to grade as product from group papers (estudy 15) 16. Online courses, if they use the technologies well, can provide opportunity for learning within a communal group (estudy21) 17. social negotiation of meaning thru asynchronous/synchronous collaborative tools (estudy24) (estudy02) 18. provide forums that are private for groups of 3-4 (estudy 17) Sub-Themes Environment Pedagogy . 1. critically merge own vision with disciplinary insight and experience (estudy04) 258 2. accommodate diversity (estudy07) (estudy09) 3. create realistic expectations (estudy07) 4. act as a guide/coach/facilitator without being too prescriptive or unstructured (estudy07) (estudy06) (estudy09) (estudy04) 5. be responsive to learners needs as they arise (estudy07) 6. Consider the maturity of the students (estudy25) 7. respect the autonomy of the individuals involved as they engage in a negotiation of meaning with others (estudy25) 8. challenges and scaffolds so that their students can experience phenomena (estudy31) Structure 9. structure is appropriate - but it needs elasticity (estudy21) (estudy09) 10. provide guidelines to success (estudy 15) 11. Use tools (software, web applications) to provide guidance (estudyl3) 12. Have tools to encourage critical thinking and active feedback (estudy08) (estudy04) (estudy06) 13. adapt the learning system to the learner's different levels of abilities or preferences (estudy08) 14. encourage development of new tools and resources (estudy06) Atmosphere 15. Create and maintain an effective and safe environment (estudylO) (estudyl4) (estudy02) (estudy09) 16. allows for thinking/reflection instead of just listening (estudyl7) (estudy36) (estudyl7) (estudyl6) 17. Interesting and engaging (estudy31) 18. Learner-centered (estudy23) 19. motivational (estudy36) 20. co-operative (estudy36) 21. ethnically and culturally diverse (estudy09) 22. essentially encourages/places value on the development of personal understanding/knowledge construction by the learner (estudy22) (estudy24) (estudyl2) (estudy23) (estudy24) (estudy06) (estudy09) (estudy 16) (estudy 10) 23. an understanding that there are multiple ways that 'knowledge' is absorbed and constructed (estudyl8) 24. Students encouraged to represent their ideas in multiple modalities and from multiple perspectives , (estudy31) Activities 25. activities/experiences such as problem-based learning approaches, collaborative learning opportunities, open-ended learning opportunities, self-paced learning opportunities, and varying pathways through the subject matter (estudy21) (estudy36) (estudy 12) 26. learning activities that stretches students to learn more but does not overwhelm (estudy05) (estudy08) (estudylO) 27. Provide good set of resources (estudy 14) (estudy26) (estudy08) (estudy22) (estudy06) (estudy09) (estudy06) 28. Create and manage or implement media-rich opportunity for experiences (estudy09) (estudy21) (estudy35) (estudy06) (estudy09) (estudy04) 29. active or higher-level thinking (i.e., increased effort) (estudy36) 30. individual/independent content/concept mapping (estudy36) 31. high levels of purposeful activity, and often toward authentic goals decided by learners or negotiated with them (estudy25) (estudy 12) (estudy31) (estudy08) 32. tasks should be based in a realistic mix between knowledge of the world and knowledge of the discipline (estudy3 1) 33. opportunities for micro and macro exploring either as individual or group (estudy09) 34. Prior learning is key - setting upon which and the tools by which further learning occurs (estudyOl) (estudy 15) 259 35. active and meaningful interaction with the learner's real environment (Akhras and Self, 2000) (estudy08) Assessment 36. Monitor and assess student learning via authentic assessments, that is, projects and problem-solving tasks resembling those encountered outside of the classroom (estudy06) 37. promote a number of different ways to achieve/apply/assess the learning (estudy25) (estudy29) (estudy09) (estudy36) Features/Characteristics 38. Collaboration on tasks, shared interpretation and dialogue contribute to the creation of a community of learners (estudy23) (estudy21) (estudy25) (estudy03) (estudy36) (estudy31) (estudylO) (estudyl2) (estudy06) 39. in-depth analysis (estudy36) 40. self-experiential referencing (i.e., relevant and meaningful) (estudy36) (estudy09) (estudy 16) 41. inquisition or investigation (estudy36) 42. research-oriented (estudy36) 43. sensitive to context and its importance in influencing learning (estudy25) (estudy08) 44. emphasise active participation of learners with the content and with each other, with the intent of helping build a high degree of relevance to individuals and groups (estudy25) (estudylO) (estudyl2) 45. multi-modal, rich, varied and multi-faceted (estudy25) (estudy37) (estudy22) (estudy06) (estudy09) 46. Bransford, Brown, & Cocking conclude that learning *environments* should be: Student-centric; Knowledge-centric; Assessment-centric; Community-centric. (estudy33) 47. Socially-mediated (estudy 10) 48. is practical (estudy37) 49. opportunities/facilitation abductive, multiloguing about the results of the exploration (estudy09) 50. People are always learning through various activities, educational and informal (estudy06) (estudy08) 51. the process is iterative and lifelong so that our own "meaning-making" is constantly changing (estudy02) 52. progress from one stage of development to another over time (Vygostky's zone of proximal development) (Akhras and Self, 2000) (estudy08) Sub themes - Instructors Leadership 1. lead and follow (estudy 15) 2. step aside and allow the class to figure out how it will go down the path that is established (estudy 15) (estudy20) (estudy23) (estudy02) Preparation 3. lay down the groundwork/provide the frame of reference (estudy 15) 4. create appropriate constructivist assignments (estudy22) 5. responsible for the course resources (including physical or human - peer, technical support) (estudylO) (estudy24) (estudy05) (estudy08) 6. select the major learning outcomes that the instruction is supposed to foster (estudy26) 7. Provide relevant examples (estudyl6) (estudy02) 8. space for dialogue about social and contextual issues, and even chit-chat, as well as 'on-task' interaction(estudy 18) 9. Learning activities value need to allow for reflection, interaction, critique and time to generate new learning/self-reflection (estudy25) (estudy03) (estudy37) (estudy23) (estudy24) 10. ability to use the tools, processes and procedures(estudy04) 260 Strategies 11. Provide opportunities not always answers (estudy09) 12. not primarily a transmitter of information(estudy21) 13. avoid direct instruction (estudy24) (estudy37) 14. watch for the 'teachable moment' and intervene (estudy07) (estudy26) 15. support multiple conceptions and promote critical thinking that stretches boundaries(estudy23) (estudy34) (estudy 16) (estudy02) (estudy 18) (estudy05) (estudy08) (estudy 10) (estudy02) 16. learning strategies that allow for authentic learning and new experiences (estudy02) 17. strategies for learners to work effectively in online learning groups (estudy20) (estudy05) (estudy 17) (estudy 16) 18. facilitate learners'active participation/dialogue/interaction (estudy03) (estudy34) (estudyl4) (estudy29) (estudy05) (estudy 16) (estudylO) 19. flexible to respond to different starting points for different individual learners (estudy37) (estudy08) (estudyOl) (estudy29) (estudy03) 20. provide opportunities for learners to have control and choice within the parameters of the course (estudy23) (estudy03) 21. flexible in judging outcomes/success of learning (estudy37) Presence 22. have a good online presence (estudy05) 23. foster a positive/safe attitude/environment (estudy05) (estudyl4) (estudy02) (estudyl7) 24. responsible for the social mileu (estudylO) (estudy20) 25. authentic help to assist students achieve their goals (estudyl3) (estudy31) (estudy08) (estudy03) 26. open, encourage and maintain clear lines of communication/dialogue (estudy06) (estudy23) (estudy37) (estudy 16) (estudy 17) (estudy 16) 27. uses proactive intervention (with a spirit of caring vigilance) (estudylO) 28. kind (estudy 16) (estudy02) 29. encourage the social construction of knowledge (estudy23) 30. encourage voices not only from the dominant culture1 (estudy 16) 31. Value individual experience (estudy07) 32. Responsive (estudy 18) Challenges 33. personalize the online classes (estudy05) 34. lack of training before teaching online (estudy05) 35. need to understand the 'science of mind' before they can practice the art of good teaching or instructional design (estudy07) 36. need to learn the pedagogy of their discipline (estudy07) (estudy06) 37. Need to relinquish power and certainty (estudy25) 38. Need do a deep values inventory on themselves (estudy25) 39. need to put some of our own most treasured orthodoxies on hold (estudy25) 40. need to integrate the cognitive (thinking skills) and the affective (e.g., motivation) to be effective (estudylO) 41. to weave a delicate thread through personal experiences/perspectives, course content, theory, and discussion (estudy34) 42. shift to learner-centered philosophy (estudy23) ( • Titles/Descriptors 43. Coach, Guide and facilitator (estudy22) (estudy09) (estudy07) (estudy06) (estudy25) (estudylO) (estudy03) (estudy34) (estudyl3) (estudy20) (estudy33) (estudy36) (estudyl2) (estudy29) (estudy26) (estudy31) (estudy 16) 44. mediator, monitor (estudy24) (estudy06) (estudy36) (estudy37) (estudy08) 261 45. knowledge expert/resource who can provide different levels of knowledge and expertise for undergraduate and graduate students (estudy05) (estudy06) (estudy03) (estudyl3) (estudy36) (estudy29) (estudy26) (estudy23) 46. consultant (estudy 17) (estudy 13) (estudy36) 47. moderator (estudy07) (estudy36) 48. cheerleader (estudy25) 49. negotiator (estudy25) 50. team leader(estudy25) 51. provocative questioner (estudy25) (estudy 14) 52. learning environment designer(estudy25) (estudy 12) 53. Co-Learner (estudylO) (estudy20) (estudy23) 54. Model for effective strategies (estudy03) (estudy20) (estudy 14) (estudy29) 55. advisor (estudy 13) (estudy36) 56. manager/project manager (estudy33) (estudy36) 57. Learning agent (estudy36) 58. critical friend (estudy33) (estudyOl) 59. Learning Architect (estudy36) 60. Mentor (estudy36) 61. "Networking Node" (estudy36) 62. liaison (estudy36) 63. "devil's advocate" (estudy36) 64. counsellor (estudy36) 65. director (estudy36) Sub-Themes - Learner Prior Experience/Knowledge 1. The extent to which an educator will provide a recipe of guidelines for their students often depends on the academic level or experience of the students they are dealing with (estudy22) 2. Share/draw on relevant work & research experiences/knowledge (estudy05) (estudy35) 3. Learning built on prior knowledge (estudy34) (estudy20) (estudy23) (estudy07) (estudy24) (estudylO) (estudy3 7) (estudy3 5) (estudy26) 4. learners individually interpret their experiences and build their own internal representations of knowledge (estudy24) 5. requires a degree of self-knowledge and awareness on part of learners (estudy37) 6. have some understanding of constructivist methodology in order to develop appropriate skills and become successful as more active, self-reflective learners (estudy29) Responsibility 7. opportunity to design and carry out one's own learning agenda (estudylO) (estudy37) 8. take responsibility for own knowledge construction (estudy20) (estudy25) (estudyl2) (estudy03) 9. develop an understanding of possible solutions or responses, as they work individually or collaboratively(estudy20) (estudy25) (estudy 12) (estudy02) 10. share information and growing knowledge with colleagues(estudy20) (estudy 12) 11. take steps towards thinking and acting appropriately in the discipline or skill area (estudy03) 12. engaged in critical thinking/reflection (estudy23) (estudy20) (estudy 18) 13. learning is intentional, personal, and useful for the learner (estudy20) 14. learners are active as individuals and as group members (estudy20) (estudy37) (estudy04) (estudy25) (estudy 12) (estudylO) (estudy24) (estudy 18) 15. seeks meaning through constructive processes - forming, elaborating, and testing mental structures until one makes sense to him/her (estudy21) (estudy25) (estudy03) (estudy07) (estudy06) (estudyl8) 16. knowledge is not transferred, but is constructed by the learners (knowers) (estudy04) (estudy25) (estudy 18) (estudyOl) (estudy21) (estudy03) (estudy05) (estudy03) (estudy07) (estudy06) 17. learning takes time and requires motivation (estudylO) (estudyl3) 262 18. Learning is a social process (estudy 14) (estudy20) 19. influenced by teaching and perceived expectations as well as by various aspects of the social context it takes place within (estudy07) 20. an understanding that learning and new understandings can develop using various tools and approaches - applied and theoretical (estudy 18) 21. be considered in the context of authentic tasks, communities of practice and engaged in learning as a social and dialogical process (estudy29) (estudyl4) 22. messy and complex and can be approached from multiple perspectives (estudy26) Instructor 23. encourage ownership/proaction in learning (estudy21) (estudy03) (estudyl7) (estudy02) 24. self-regulation of the learning environment (estudy21) 25. regular contact with the learners (estudy04) (estudy37) 26. work guided by learners needs (estudy04) (estudy06) 27. support learners by modelling the role of a problem solver (estudy24) (estudy07) 28. Arrange online instruction to meet individual learner needs (estudy21) 29. establish trusting relationships with students (estudy06) (estudy 16) (estudy02) 30. Support students to negotiate technical and performance requirements (estudy06) 31. use scaffolding techniques that support the learner (estudy24) (estudy20) (estudy 14) (estudy26) (estudy08) 32. encourage maturity through helping students be more self-directed and self-regulated (metacognition) in their learning (estudy05) 33. offer appropriate/timely feedback (estudy05) (estudy 16) 34. encourage students to share, discuss, and (critically) reflect on what they are learning (estudy20) (estudy26) (estudy23) (estudy23) (estudy08) (estudyOl) 35. support students diverse levels of knowledge (estudy 14) (estudy23) 36. offer opportunities for students to follow their own interests (estudy 14) 37. Works with the learners to establish criteria for evaluation of their learning (estudy 12) 38. Skilled at enabling students (estudy37) 39. Flexible as different students will move in different ways and at different speeds (estudy37) 40. encourage learners to assess their work themselves and to present to other learners to obtain feedback(estudy08) 41. manage group dynamics so that collaborative work will be meaningful and useful without learners getting on each other's nerves or some learners continuing to be silent and some learners continuing to be domineering (estudy08) (estudy02) 42. train student leaders to facilitate discussions (estudy08) (estudy02) 43. assign roles and ensure learners take turns while.discussing (estudy08) (estudy02) 44. monitor and respond to ongoing concerns and needs of different students (estudy06) (estudy35) (estudy08) (estudyOl) 45. challenges/assists learners to see links between and among multiple conceptions/perspectives (estudy23) (estudy03) (estudy34) (estudy20) (estudy26) (estudy31) (estudy08) (estudyOl) (estudy02) 46. respect that learners will create valuable knowledge (estudy25) 47. works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, se)f-actualized learners (estudy25) (estudyl2) (estudyOl) 48. help the learner become more independent and successful as a learner generally (e.g., learn to manage their time, to learn how to access resources such as the library and develop research skills) (estudy03) (estudy 13) 49. shows learners how they have moved forward (estudy 37) 50. allows sufficient time within the course for students to work collaboratively (estudy20) Activities 1. experiential learning (estudy09) (estudy05) 2. role-play scenarios where students research their roles (estudy09) •: 263 3. group/open ended discussions/dialogue (estudy04) (estudy07) (estudylO) (estudy34) (estudy09) (estudy26) (estudy23) (estudy36) 4. learning-by-doing/activity based learning(estudy04) (estudy20) 5. WebQuests (estudy04) (estudy09) (estudy 14) 6. critical inquiry (estudy04) (estudy34) (estudy20) 7. problem based/inquiry based learning (estudy24) estudy04) (estudy22) (estudy09) (estudy03) (estudy 18) (estudy20) (estudy 14) (estudy34) (estudy26) (estudyOl) 8. authentic intellectual work (estudy24) 9. authentic learning tasks(estudy03) (estudy31) (estudy02) 10. motivating assignments/tasks(estudy24) 11. Simulations (estudy05) (estudy07) (estudy34) (estudy26) (estudy36) (estudy02) 12. reflective individual writing assignments (estudy05) 13. Evidence based practice (estudy07) 14. working models (estudy07) 15. interactive multimedia (estudy07) (estudy34) 16. Collaborative construction of common projects or representations (estudy06) (estudy20) (estudy34) (estudy26) (estudy31) (estudyO 1) (estudy02) 17. Small-group inquiry exercises, reported out to the larger class(estudy06) (estudy02) 18. Discussion and reflection around team activities(estudy06) 19. Threaded discussion of assigned content or study questions(estudy06) 20. guided discovery learning approaches (estudy03) (estudy35) (estudy36) 21. independent reading, whether online or offline - framed as reading as self-guided R E S E A R C H (estudy 18) (estudy02) 22. case-based approach (estudyl8) (estudy34) (estudyl4) (estudyOl) (estudy36) (estudy02) 23. e-portfolios (estudyl3) 24. project-based learning (estudy20) (estudy 14) (estudy34) 25. resource-based learning (estudy20) 26. co-operative learning (estudy20) (estudy 14) (estudy26) (estudy36) 27. Computer mediated communication (estudyl4) 28. Self-directed learning (estudy34) 29. synchronous online sessions *if* they involve exchange and discussion (not online lectures) (estudy26) 30. research papers and presentation of findings (estudy36) 31. peer teaching (estudy36) 32. content/topic development (estudy36) 33. content/topic resource gathering (e.g., "scavenger hunt") (estudy36) 34. Gaming (estudy36) 35. Debates (estudy02) Assessment 36. peer assessment (estudyl5) (estudy07) (estudylO). 37. student self assessment (estudy24) (estudyl5) 38. portfolio assessment(estudy24) 39. teacher assessment of students (estudy07) 40. Include methods for demonstrating learner skill/competence/depth of understanding (estudylO) 41. all learning activities, problems and assessments need to relate to intended objectives and outcomes(estudy03) 42. learning can have aims but these cannot be too structured ie not competency or right/wrong assessment/evaluation techniques(estudy37) 43. activities in which learners engage should be learner centered and measured against outcomes (estudy23) Theory 1. 'common sense' theory of learning (estudyl5) 2. descriptive theory or as a prescription for how to teach (estudy06) 264 3. not a "theory of learning" at all, but rather a philosophy of learning (or more accurately, an epistemological perspective) (estudylO) 4. not *a* theory of learning but a descriptive term for a series of similar, but not always mutually compatible, theories of learning (estudy26) 5. Knowledge does not exist independent of the learner (estudy21) 6. belief in multiple realities and multiple perspectives (estudy21) 7. it is important to not relegate constructivism to a single theory - it takes many forms and is based in many philosophical stances (estudy21) 8. theory based on an assortment of academic disciplines and ideas that has been formalized in the educational area (estudy05) 9. Knowledge is unique to the individual, and it is created by the individual in a peculiar way, so it calls into question whether any two learners can share the same knowledge (estudy25) 10. distinction between cognitive constructivism, where learners experience some kind of dissonance, and engage in cognitive activities to make sense of the world, and socio-cultural constructivism, which emphasizes that learning is embedded in social and cultural practices (estudy25) 11. a philosophy that draws on constructivist tenets embraces the idea that learning is created and learners may take away more from the experience that originally anticipated (estudy23) 12. based on a weak subjectivist epistemology, in which the focus is on how the individual internally constructs meaning. A possible but not necessary consequence of premising learning on subjectivism poor rigour or mechanisms for evaluating learning. Thus the criticism that constructivism can lead to 'content-free learning' where the focus is on feelings (estudyOl) 13. differences within the schools of constructivism about the importance of social interaction versus individual cognition and our ability to know what is real(estudy02) 14. assumes a subjectivist view of the world, even a post-modem one of multiple realities and views and no "grand narratives" (estudy02) 15. theory of real learning mostly centred on activities outside of courses, classrooms and teachers (estudy 13) 16. One risk is that it can lean toward relativism: anyone can build his/her own knowledge, but there is no "truth". Social version: we agree on some knowledge construct, disregarding its accuracy (estudy04) 17. the process is subjective and cognitive and recognizes that knowledge is ambiguous and truth uncertain (estudy02) 18. there is no 'reality', there are multiple realities and no right or wrong. Learning is constructed as a result of the process/es where multiple perspectives are revealed, discussed, then reconstructed (estudy34) (estudy21) 265 APPENDIX I ROUND 3 F U L L D A T A SET Statement Agree/ Disagree Rank (1 -essential - 5-not important) View A I) 1 2 3'- 4 ~ 5 Personal Global Encourages/values the development of personal understanding/knowledge construction by the learner. (N=33) 100% (n-33; 0%(n-0) 64% (n=21) 27% (n=9) 9% (n=3) 0% (n=0) 0% (n=0) • 9% -(n=3) 91% (n=30) The core of the design is learner-centred. (N=31) 100% (n-31) 0%(n-0) 42% (n=13) 52% (n=16) 3%(n=l) 3%(n=l) 0% (n=0) 16%(n=5) 84% (n-26) The process is iterative and lifelong so that our own "meaning-making" is constantly changing. (N=33) 97% (n-32) 3%(n-l) 33% (n=ll) 33% (n=ll) 30% (n=10) 0% (n=0) 3%(n=l) 15%(n=5) 85% (n-28) Understanding that there are multiple ways that 'knowledge' is absorbed and constructed. (N=33) 97% (n-32) 3%(n-T) 33% (n=ll) 48% (n=16) 18% (n=6) 0% (n=0) 0% (n=0) 24% (n=8) 76% (n-25) Supports multiple conceptions and promote critical thinking that stretches boundaries. (N=32) 97% (n-31) 3%(n--|) 41% (n=13) 28% (n=9) 22% (n=7) 9% (n=3) 0% (n=0) 28%(n=9) 72%. (n-23) Fosters active/higher-level thinking and reflection. (N=33) 94% (n=31) 6% (n-2) j 3 % (n=ll) 52% (n=17) 15% (n=5) 0% (n=0) 0% (n=0) 24% (n=8) 76% (n-25) Emphasise/facilitate active participation of learners with the content (dialogue) and with each other (interaction), with the intent of helping build a high degree of relevance to individuals and the group. (N=33) 94% (n-31) 6% (n-2) 45% (n=15) 18% (n=6) 33% (n=ll) 0% (n=0) 3%(n=l) 27% (n=9) 73% (n- 24) Learners are encouraged to seek knowledge and experience from different types of sources and perspectives. (N=33) 91%, 01-30) 9%(n-3) 33% (n=ll) 36% (n=12) 24% (n=8) 6% (n=2) 0% (n=0) 15%(n=5) 85% (n-28) Sharing multiple perspectives and hearing the constructions of others allows one to question our original assumptions and to develop a shared meaning. (N=33) 91% (n-30) 9%(n-3) 36% (n=12) 27% (n=9) 24% (n=8) 9% (n=3) 3%(n=l) 36% (n=12) 64% (n-21) Learning is constructed as a result of the process(es) where multiple perspectives are 91% (ir=30) 9%(n-3) 21% (n=7) 30% (n=10) 36% , (n=12) 9% (n=3) 3%(n=l) 39% (n=13) 61% (n=20) 266 revealed, discussed, then reconstructed. (N=33) Provides opportunities, but not always answers. (N=32) ' 91% ' (n=29) 9% (n=3) 25% (n=8) 31% (n=10) 44% (n=14) 0% (n=0) 0% (n=0) 41% (n-13) ' 59% • (n=19) Offers opportunities for students to follow their own interests. (N=32) 91% • "(n=29) 9% (n=3) 28% (n=9) 25% (n=8) 38% (n=12) 9% (n=3) 0% (n=0) 50% (n-16) 50% -(n=16) Learners should have opportunities to solve complex, realistic problems similar to those that the learners themselves would encounter outside the classroom. (N=33) >  88% (n=29) 12% (n 4) 24% (n=8) 30% (n=10) 30% (n=10) 15% (n=5) 0% (n=0) 39% (n=13) 61°/o (n=20) Uses a learner-centred approach to teaching, which involves sharing authority with learners. (N=32) ". 87% -> (n=28) 13% (n-4) 22% (n=7) 44% (n=14) 22% (n=7) 13% (n=4) 0% (n=0) 41% (n-13) 59%, >=19) . Works to nudge students toward the kind of intellectual responsibility they need to become self-motivated, self-actualised learners. (N=32) <' 87%' * (n=28) 13% (n-4) 34% (n=H) 38% (n=12) 22% (n=7) 6% (n=2) 0% (n=0) 47% (n-lS) 53% (n=17) ' Encourages the voices of all learners and is respectful of what the learner can contribute to the learning enterprise. (N=31) ' 87% ' (n=27) 13°,. (ir 4) 23% (n=7) 32% (n=10) 39% (n=12) 3% (n=l) 3% (n=l) 55% (n-17) 45% (n~14) Assessment should avoid overly narrow measurement of discrete skills or knowledge without a more complete context of interpretation. (N=31) 87% (n=27) : 1 V > 0 (n=4) 19% (n=6) 55% (n=17) 19% (n=6) 6% (n=2) 0% (n=0) 48% (n-15) 52% (n-16) Open-ended discussions based on critical inquiry/authentic intellectual work. (N=33) - 8 5 % J(n=28) . 15% in-5) 9% (n=3) 33% (n=ll) 42% (n=14) 12% (n=4) 3%(n=l) 48% (n-16) 52% in=17) . Builds skills and abilities while working on authentic tasks and problems. (N=33) : 85% • • (n=28) 15% •:(n=5) 12% (n=4) 33% (n=ll) 42% (n=14) 12% (n=4) 0% (n=0) 36% (n-12) 64% >(rir24): . r N —— • Provides and balances self-directed and . collaborative learning opportunities. (N=32) 84% -\i (n=27)% 16% (n-5) 13% (n=4) 22% (n=7) 53% (n=17) 13% (n=4) 0% (n=0) 56% (n-18) 44% "(n=14)- " Seeks engagement with a community of learners or practice. (N=32) 84%. ' (n=27) 16% (n-5) 22% (n=7) 22% (n=7) 41% (n=13) 9% (n=3) 6% (n=2) 38% (n-12) 62% (ir--20) Becomes a metacognitive coach for learners. (N=32) 84% - (n=27). 16% • (n=5) 13% (n=4) 41% (n=13) 25% (n=8) 19% (n=6) 3%(n=l) 56% (n-18) '44% (n=14) The instructor is flexible enough to support student expressions of different levels of knowledge. (N=31) 84% (n=26) 16% . (n=5) 19% (n=6) 42% (n=13) 29% (n=9) 6% (n=2) 3%(n=l) 58% (n-18) 42% -(n=13), Should create an environment with a social > -84% 16% 26% 26% 26% 23% 0% (n=0) 45% ' > 55% 267 context. (N=31) (n-26) (n-5) (n=8) (n=8) (n=8) (n=7) (n=14) (n=l7) Assessment should often include learners' reflections on their goals and accomplishments. (N=31) 84°,, (n-26) 16% (n-5, 16% (n=5) 45% (n=14) 26% (n=8) 13% (n=4) 0% (n=0) 68% (n=21) 32'!., (n-10) Builds on prior learning (knowledge) of individuals. (N=33) 82% (n 27) 18% tn 6) 33% (n=ll) 30% (n=10) 27% (n=9) 6% (n=2) 3%(n=l) (n=ll) 67% ;.(ii=22) Critically merges the learner's knowledge of the world with disciplinary insight and experience. (N=33) 82%, in-27) 18"., (n 6) 18% (n=6) 36% (n=12) 36% (n=12) 6% (n=2) 3%(n=l) 36% (n=12) 64%, (n=21) Emphasises experiential, authentic learning, which is decided on by learners or negotiated with them. (N=33) 82% (n-27) 18% (n- 6) 24% (n=8) 24% (n=8) 27% (n=9) 15% (n=5) 9% (n=3) - 52% (n=17) 48% -•(n=l-6) Encourages exploration of multiple paths and forms and facilitates opportunities for varying experiences and pathways through the subject matter. (N=32) 81% (n-26) HMHi 19% (n-6) 22% (n=7) 19% (n=6) 38% (n=12) 22% (n=7) 0% (n=0) . 53% • (n=17) 47% (n-15) Social negotiation of meaning can happen through asynchronous / synchronous collaborative tools. (N=32) 81% 1 19% (n-26) , (n-6) 19% (n=6) 22% (n=7) 34% (n=H) 16% (n=5) 9% (n=3) 44" o (ir 14) 56% fn-18) Assessments should accommodate the needs of diverse learners. (N=31) 81% (n-2S) 19% ( i 6) 16% (n=5) 35% (n=ll) 35% (n=ll) 13% (n=4) 0% (n=0) 58% (n 18) 42% (n-13) Learners may take away more from the experience than originally anticipated. (N=31) 81% (n-25) 19% (n-6) 26% (n=8) 16% (n=5) 26% (n=8) 26% (n=8) 6% (n=2) 61% (n 19) 39% (n-12) Support the goals of learners by facilitating appropriate forms of learner collaboration and discussion. (N=32) , 78% ,• (n-25) 22% (n-7) 6% (n=2) 25% (n=8) 38% (n=12) 28% (n=9) 3%(n=l) 69% (n=2'2) 31% (n-10) Flexible in responding to different starting points for different individual learners. (N=32) 78 (n-25) ' %••• (n-7) 19% (n=6) 41% (n=13) 31% (n=10) 6% (n=2) 3%(n=l) 56% (n 18) 44% (n-14) Learning activities are "authentic" and require reflection. (N=33) 76 (n-25) 24% (n-8) 18% (n=6) 24% • (n=8) 45% (n=15) 9% (n=3) 3%(n=l) 55"o (n 18) 45% (n-15) Knowing is mediated by community/culture and the tools of that community and culture, so be aware of and exploit cultural differences. (N=32) 75%, (n-24) 25% 13% (n=4) 16% (n=5) 53% (n=17) 6% (n=2) 13% (n=4) 69% (n 22) 31% (n=10) Provide activities at the right level of challenge for learners. (N=31) 74% (n-23) 26° o (n-8) 16% (n=5) 39% (n=12) 32% (n=10) 6% (n=2) 6% (n=2) 42%> (n- 13) 58% -'(n=18) Should accommodate a shift in expected learning 74% 26% >' 13% 29% 32% 16% 10% 6.- • 35% , 268 outcomes. (N=31) (n=23) (n=8) (n=4) (n=9) (n=10) (n=5) (n=3) (n-20) (n=H) Provides opportunities for dialogue between novices and experts. (N=31) , 74% ' (n=23) 26% (n-8) 19% (n=6) 23% (n=7) 26% (n=8) 29% (n=9) 3%(n=l) 58".. (n-18) 42% (n=13) Assessment systems should examine a learner's full range of knowledge, skill, and attitudes toward a domain of expertise. (N=31) 74% (n=23) 26".. (n-8) 23% (n=7) 29% (n=9) 26% (n=8) 13% (n=4) 10% (n=3) 68% (n-21) 32% (n=10) Learners should have access to assessment criteria and methods, and be able to negotiate at least some of the terms of the assessment. (N=31) 74% (n=23) 26% • (n-8) 26% (n=8) 32% (n=10) 23% (n=7) 16% (n=5) 3%(n=l) 61% (n-19) • . 39% V (n=12) Knowledge does not exist independent of the learner. (N=33) - 73% (n=24) 27% in-9) 42% (n=14) 24% (n=8) 21% (n=7) 9% (n=3) 3%(n=l) 24% (n-8) 76% -(n=25) Learners interact with a media-rich and resource-rich environment in building knowledge. (N=33) 70% -' (n=23)' 30°,. (.1-10) 9% (n=3) 18% (n=6) 33% (n=H) 36% (n=12) 3%(n=l) 52% (n 17) 48% (n=16) Provides a safe environment, encouraging risk-taking and authentic dialogue. (N=33) ' 70% .(n=23) 30% ln-10) 24% (n=8) 30% (n=10) 30% (n=10) 9% (n=3) 6% (n=2) 2% (n=l7) • 48%-. • (n=16) Emphasizes co-operative learning (collaborative construction of common projects or representations). (N=32) • 69% -(n=22) ' 31% (n-10) 3%(n=l) 34% (n=ll) 25% (n=8) 34% (n=H) 3%(n=l) 56% (n-18) 44% (n-14) Constructivism assumes a subjectivist view of the world, even a post-modern one of multiple realities and no "grand narratives." (N=32) /- 66% , (n=21) 34% «n- l l ) 13% (n=4) 22% (n=7) 28% (n=9) 22% (n=7) 16% (n=5) 56% (n--l8) .*% (n=14) Provide a variety of ways of introducing course material so those students with different learning styles are accommodated. (N=31) 65%' (n=20) ' 35% (n-11) 13% (n=4) 16% (n=5) 42% (n=13) 19% (n=6) 10% (n=3) 61% (n 19) 39",, (n-12) Provides a clear description of the rules of the course so that all learners understand the playing field. (N=31) 61% (n=19) 39% (n 12) 10% (n=3) 45% (n=14) 23% (n=7) 10% (n=3) 13% (n=4) 52% (n-16) 48"., (n=15) Steps aside and allow the class to figure out how it will go down the path that is established. (N=32) 44% (n=14) 56% (n-18) 6% (n=2) 22% (n=7) 25% (n=8) 34% (n=ll) 13% (n=4) 72% (n-23) 28% (n=9) Practised in the art of evidence-based teaching and instructional design. (N=32) 38% -(n=12) 62% (n-20) 16% (n=5) 3%(n=l) 31% (n=10) 28% (n=9) 22% (n=7) 72% n -23) 28% (n=9) A variety of learner performances and products (both individual and group-based) may be suitable for assessment, including: Content/topic development/resource gathering - 81% 19% 13% 32% 32% 19% 3%(n=l) 42% * 58% 269 (N=31) (ir-25) (n-6) (n=4) (n=10) (n=10) (n=6) (n=13) (n=18) Creative works (N=31) 77% (n-24) 23% (n -7) 13% (n=4) 26% (n=8) 23% (n=7) 29% (n=9) 10% (n=3) 61% . (n=19) 39% (n-12) Problem-based learning (N=31) 77% (n=24) 23% (n-7) 16% .(n=5) 39% (n=12) 23% (n=7) 23% (n=7) 0% (n=0) 52% (n=16) 48% lnH5) Case-based learning (N=31) •77% (n=24) 23% (n=7) 10% (n=3) 32% (n=10) 32% (n=10) 23% (n=7) 3%(n=l) 52% ~ (n 16) 48"» (n-15) Asynchronous discussion (N=31) 74% (n=23) 26% (n=8) 23% (n=7) 19% (n=6) 39% (n=12) 13% (n=4) 6% (n=2) 45% (n=14) 55% . (n'=17) Portfolios/e-portfolios or collections of work samples (N=31) 71% ln-22) 29% (n=9) 13% (n=4) 32% (n=10) 26% (n=8) 26% (n=8) 3%(n=l) • .52%, • (n=16) 48% (n-15) Guided-discovery learning (N=31) 68% (n-21) 32°-;. (n-10) 23% (n=7) 23% (n=7) 26% (n=8) 26% (n=8) 6% (n=2) 58% (n=18) 42% (n-13) Evidence-based practice (N=31) 68% (n-21) 32% (n-10) 10% (n=3) 23% (n=7) 29% (n=9) 26% (n=8) 13% (n=4) <>1% (ii 19) 39% . (n-12) Webquests(N=31) 68% 32% (n-21) i (n-10) 3%(n=l) 13% (n=4) 35% (n=ll) 35% (n=ll) 13% (n=4) 61% (n=19) 39°/,, (n-12) Independent reading framed as self-guided research (N=31) 68% ' 32% (n-21) : (n--10) 13% (n=4) 26% (n=8) 35% (n=ll) 23% (n=7) 3%(n=l) 65% .(n=20) 35% (n -H) Live performances (N=31) 65% (n-20) 35% (n=ll) 3%(n=l) 13% (n=4) 32% (n=10) 29% (n=9) 23% (n=7) 68" 0 (n 21) 32% (n- 10) Inquiry reports (N=31) 65% (n-20) 35% (n=ll) 10% (n=3) 23% (n=7) 16% (n=5) 29% (n=9) 23% (n=7) 58";. (n=18) 42% (n-13) Performance assessments and skill demonstrations (N=31) 65% (n-20) 35% (n-11) 16% (n=5) 13% (n=4) 32% (n=10) 29% (n=9) 10% (n=3) . 52% (n=16) 48% (n-15) Small group inquiry exercises, reported out to the whole class (N=31) 65% (n-20) 35% (n=ll) 13% (n=4) 29% (n=9) 32% (n=10) 19% (n=6) 6% (n=2) 55",. (n 17) 45% (n=14). Research papers and presentation of findings (N=31) 65% (n-20) 35% (n=ll) 13% (n=4) 23% (n=7) 29% (n=9) 32% (n=10) 3%(n=l) . 61% in- 19) 39% (n=12) Peer teaching (N=31) 65% (n-20) 35% (n=ll) 6% (n=2) 26% (n=8) 29% (n=9) 32% (n=10) 6% (ri=2) 71% (n 22) 29% (n=9) Simulations (N=31) 65% (n-20) 35% (n=H) 6% (n=2) 23% (n=7) 23% (n=7) 35% (n=H) 13% (n=4) 71% (n=22) 29% (n=9) Role-playing (gaming) (N=31) 61% (n-19) 39% (n=12) 6% (n=2) 23% (n=7) 26% (n=8) 29% (n=9) 16% (n=5) "- 77% (n=24) 23% (n=7) Working models (N=31) 58% 42% 6% (n=2) 16% 29% 39% 10% - 65% 35% 270 (n=18) (n=13) (n=5) (n=9) (n=12) (n=3) . (n=20) . (n=l l ) Synchronous on-line sessions (N=31) , -52% (n=16) 48° •> * (n=15) 6% (n=2) 3%(n=l) 26% (n=8) 39% (n=12) 26% (n=8) 68% <n-21) 32% (n=10) 4. The numbers within each statement vary: 33 participant- experts began the Round, while some items were completed by 32 or 31 participant-experts. 5. Because of rounding of the initial numbers, some totals are 99% or 101%. 6. As in the survey, the data regarding possible activities flowing from constructivism remain separate from the main statements/beliefs. 271 APPENDIX J SCREEN CAPTURES OF STUDY WEBSITE Website Home Page: ri? Cm«Vrocti*B«W • HdWtdyt Welcome to Ihis online atudy ConatruclM&m and In&tioctionat Design: An Bjiorsiw Using BnAsyrKfirQnous Online Homme! Group Technique. The purpose of/this study is to identity what you as participant-e*per!s believe art I lie tenets/pre sup posit ion 9 of con stw Urn sm asa theory urieamjng, end to suggest instructional approaches that .(low from Ihis theory for (he development ofonhne learning Using a consensus-huiiaing methodology, the Nominal Group Technique (NGT), wholly online, the goal is to reach a consensus among the participants on what constructivism means and its implications for the design of online reaming programs. You have bean aeked lo participate 11 this research because otyouraxpertiM and experience in the area ofteaming theory, educational technology andfor online learning. Constructivism - w> ' I n s t r u c t i o n a l D e s i g n Please review the irrformallon underthe Confidentiality'. 'Contacts' and Research Teem' links before beginning. They can be found al the lop of each page. The Home' link brings you back to the homepage To begin the study, click on the link below iha Nntiiln,il tirmip Ier.hnlqui> Introduction Page: ;' l«TT«cir-ttiijiw -Hewc*w ..':rr:.:'" Nominal Cioup Technique This study wiK use en online version of the Nominal Group Technique (NGT), a methodology of consensus building. The overall intention of the NGT is to generate ideas individually end anonymously, which are then discussed and rankec Dy the group Specific details on the full process will be supplied once you agree to participate Constructivism — I n s t r u c t i o n a l D e s i g n HOWM Proceed As each round is released (and you racarv* an email notifying you olthe Beginning of each round), you are asked to click on the link below ihel corresponds with the active round in the process. Rounds not in use will be Oisabisd until they are to be used and after the round is finished Please review the information under the •Confidentiality', 'Conlacts' and Research Team' Link before beginning Whan you need to. use the Sack' link on each page to return to this page. The Home' link bongs you back to the homepnge Using WebCT's Discussion Forum* We are using WebCT's discussion forum to conduct the discussions (both private and group within the Rounds) If you are femiliar with this feeture of WebCT. scroll down lo the end of this page, and link to Round t, log into the discussion area (see the login button on the right of the screen) and begin. If yDuere unfamiliar with how the WebCT discussion forur 272 Introduction Page (continued): 1 NGT TmliniutM N . W , „ Using Wo hCT's Discussion Forum* We are using WebCT's discussion forum lo conduct the discussions (bo(h private and group within the Rounds), if youers familiar wiih this feature of WebCT, scroll down to the end of I hist page, arid link to Round 1, log into the discussion area (see the login button on the right of the screen) end begin. finished the Tutorial, scroti down to the end of this page. and link to Pound 1. log into the discussion area (seethe login button on the right of the screen) and begin.: Participation How much lime will you need fcr panic ipaiion? Of course the imi needed for you lo pert op are can only be estimated Overall, lor the 4 round*, tou writ need lo set aside 1-J hows (for Round* '3) end 5 3 hours (lo> Round 4; dunng the * week* of acirvt padlCipeuon »W ttripi .itin.r Evaluation ntthf' Irtna Round 1 Page: Round 1: Brainstorming (Duration: approximate 1 week] The goal of the first round re the identification by the panicipsm-expeits of the ideas or cheractenslics that apply to the three questions and the basic theme of the research In Ihis first round, it is important thai Ihe ideas are proposed independently and without discussion wilh other participant-experls This round will be conducted by posting tn a private discussion forum provided to you tor this purpose Only you, end I. (he facilitator'rese arc her will see tins initial work Each participant wilthave his/her own private space: C&miMUCtM&m W > I n s t r u c t i o n a l D e s i g n • To begin Round 1. please click on Ihe 'Log onto WebCT- link lo the right This link will brmg you to Ihe "Welcome to WebCT page". » Type your user name and password mo the appropriate spaces Reese uee the user name and password thai were issued to you by email si the commencement of Ihe project. • After logging in you wilt see a page with an icon thai resembles a push pin. Click on litis icon end you will be brought into the discussion hoard • Clicl on yum pnvaie brainstorming specs. Click on the welcome message end poet when you are ready, the earns which you feel answer Ihe question* noted in the welcome message Remember this it a private brainstorm Your comments will only Be seen by me. the facilitator, (o lhat I can combine everyone's comments into S larger bst tot the whole group, for Ihe next round Comment as much and as oflen as you like in the timeframe for Ihis session. • I look fonwerd lo reeding your cc benefiting from your insights and e; 273 Round 2 Page: :S> Ruuitd J NuLcai* Round 2: Presentation ortht Lists of Brain stormed Ideas and Clarification Regarding these Ideas (Duration: approiriravtafy JwwHJ - )^ * k. B<«nd 2 DM co-tax.M «f ,d... *om ^ . . d I n s tTu c ti o H a I D e s i g n responses based on Ihe 3 quesltons in Round 1 will he Ijreumtri) lu ill JJ a it it: ip ant-(nouns Tli» participant who offered in Ham -ill not b here it to guarantee anonymiy. to enabl. • Vm tn.i. j object™* discussion el tha ideas during Ibis round The ffWW V*»*fS round will be used lo clenly, add lo, delete or merge any points on the fitt Tha goal hare is ID reach a common undemanding of tha content of tha list by the entire group • To begin Round 2. pleeae click on the 'Log onto WebCT link lo Ihe right Thu link will bring you to the "Welcome to WebCT page" • Type your user nam* and password into tha appropriate (pacta. Please use ihe user name and paiaword that were reeved to you by email si Ihe commencement of Ihe project • After logging In you will see a page with an Icon that nesembles a push pin. Click on this icon and you wvl be brought into Ihe discussion bo aid. • Click on the Discussion space entitled Round 3. Click on the welcome message to review the rules tor litis aeclron Agam, (hit round is a round for clarification of the Hems on the kst compiled frem ihe brainstorm. Your job m this round will be lo ask queitiona ol ctanicadon on item* unclear to you, and lo answer questions from others in Ihe group, on Ihe kern* you added to the liai Vou can alio pott additional Hems you feel ere missing, and suggest |h» rnerqer or oeenon other points The PlrtH ( Lag onto vVaoCT ] Round 2.2 Page: 2j> Ruund 2 Netscape Round 2.2: Small Group Reworking of Elements of Bralnstormed Ideas and Clarification Regarding these Ideas {Duration: approx/rnffy 1.S waaks) In Round 2 the combined lis) of ideas horn gathered reaponees based on the 3 questions in Round 1 were initially examined and modified/delelad/meiged or edded in. White ihit work wa« valuable, the list remains quits large. To make the leak more manageable end lo narrow down the items into a mora workable list, Round 2.7 is • continuation of thie work begun in Round 2 but with participanu'eipart* assigned lo smaller group* and each group focused on only I of the B elements identified In Round 1. Each small group will be assigned a private group space and aeked lo review I of the G elements (within each element there mey be up Io 35 individual paints). Thu 2.2 round will b* used to clarify, add ID, delete oi merge any points on ihe The goal here K ai lha end of Ihe discussion, the group will agree on 5-7 key, essential elements, from the liai. that define Ihe eelegory/elament being examined These key items will form the basic of the vote in Round 3. • To begin Round 2 2. pless* click on tha "Lag onto WebCT" link lo the nohl This Irnk writ bring you to 111 "Welcome to W.bCT • Type your user neme and password into the appropriate spaces Please use the user neme and password tha! were issued to you By email al Ihe commencement oi the projecl. • A*»r logging m you will see a page wBti an icon that - **JL> I n s t r u c t i o n a l D e s i g n - * - e * • 1X9 1 ? ; 5 4 274 Round 3 Page: Round 3: Evaluation of the Ideas tt>u ration; opproxbattuly I weeAJ The purpose of this round !• to give you Ihe opportunity !o evaluate and voice your opinion* on the individual idea* gathered so far. Thi* will ba accompli lived traouah a vote vii anonanatumey. • To begin Round 3. you w4H haw received an imitation by email to lake a vole on the revised list created from Rounds 1.2 and 2 2 H you click on the link within that email you will be taken to tha voting apace thai f am uelnt — HoetedSuivey com • Read the instructions on the mam page of Ihe survey and then click on the Senin Survey button N mould take between 45 minutes, and t hour to complete Ihe surrey. The survey uses a triple matrix baaed on the agreed-upon a (elements gathered from the brainstorm and clarified in Round 2 and 2 2 Click on fjejn; at Ihe bottom of each page to move lo the neat page (Iheie are a pages) When finished WJlmg. click Ihe E"d of gurvev button al the and • I look forward 10 reading your ranking of the items gathered torn Rounda 2 and 2 2 • PLEASE NOTE: you era required to anewat ALL of the question*. If you mkta clicking on one section. Che software w" Ihe neat pens ... If tine heppane: e review the page end find th - make a choke, leaving a cl space provided. ' M mil liwn to the bottom a on Next lo move to next pa Instructional Design If vou don 1 0*1 an »mat< mvit»tion pltrtac contact me mmcdite^ by em*r ic*y^;v. v ^ v i j j - / . » ; **Qp&n IS* Round 4 Page: Ruund 4 NBtreeiw Round 4: Internal Publication and Discussion of the Results (Duration; approximately 1 w/aety In this round the voting results, from Round 3 will be open for wewmg by all participant-experts Participants are asked to comment on tha results, to raise questions about any surprises/unexpected outcomes, and to explain why Ihey agree or disagree with Ihe results You will also be asked in tins Round to define youi own personal lenl of c thai is, what ia Ihe loweat % you feel conf o affirm youi agreement and/or is with the voting results of Round 3 Cm&imetmmu ~™ o*0 I n s t r u c t i o n a l D e s i g n ?%dittr pfleteaaeSeM ti-wwyJ ^ w*:.ywr. This Round ra • To begin Round 4, please clicK on the 'Log onto WebCT link to Ihe nght This link will bring you to the •Welcome to WebCT page". • Type yout user name and password into the appmpriate spaces Please use the usei name and password that were issued lo you by email at the commencement of Ihe project • Altar logging in you well sea a page with an icon that resembles a push pin Click on thie icon and you will be brought into lha discussion bean] » Click on the Discussion space entitled Round 4. • Click on Ihe welcome message to review the rules for this taction. Your tub in this round will be to comment on the results, to raise questions about any surpnsee/unexpected outcomes, end lo explain why you might agree or disagree with the result* •finally you will be asked to define consensus for ihe -aVutf 275 WebCT Discussion Homepage: W OJTIietis WebCT 3. S i a Nulttapu HMe Navigatim DJThesIS Round 2A Round 2B Round 3 Results sap I n s t r u c t i o n a l D e s i g n X DISCUSSIONS WebCT Discussion Forums: •2 0 llhnn - W.KT 1 t 1 . Kinder i j lie; Hide Navigation Round 2A Buund2B Round 3 Results DJThesis tttnw • DISCUSSIONS Select a topic to see Its messages Topi* Unread Total. S,m, A3 ; : 0 662 0 7 EffiBtc. locked 0 8 Private, locked sstjidyCj 0 9 private, locked 0 8 private, locked e^|yC5 ' 0 7 SfSSSi locked r«i-M£ • • 0 7 private, locked eim.dyi,' 0 7 private, locked vsiu.ly:^ 0 7 PTtvalf. locked ettudvOi- ; ; 0 8 trfjyatt. locked «tujyl0 0 7 private, locked astftij /: 0 4 private, locked 0 7 ptTVatc, locked esmdyO 0 7 private, locked e Silldy 1-1 0 7 private, locked esludvl' : 0 7 pnvale, locked 0 8 £WL*l'i- locked 0 7 i>nvate. locked M,to#:l§ 0 7 gtruflte, locked 0 4 KBMffi. !»cked 0 8 private, locked 276 Example of WebCT Discussion: f.-nv*k-W*MI I3 .U IKttnv U B C nm is H Coirlrsi Penel Manage Students Pejnd 2* RaunrJSB Pajn-1 3 Result. &.SJU% p. Discussion messages: Round4 Return to Ducusnon: Select topic Rounds V( Show al / ^owupjead s.noi j« s.im n.n. Apply to jeleettd metiage<i, below Statue Subject '.' ^ Cornmenti on Round 3 • Ihe Tommntti Round 3 - Ihe > 1 Reiuia • Wui-J t ..Re RiundTRr-ruft.-U',-• 0/25 0 • Id 1 n e • • ft • a 0 ft o 0 • a a n • • a II D F.e K-iei<i f Kesufts - W,, u P.f K,-id 3 ReeulB - Wo ' t i l i i iMlThi-ii TI'I tHefJL^Hr. ii i- j Threaded / i Author ertudy 0 (estudyO) ertudy 0 (ettudyO) ertudy 0! (ertudy05) eitudy 35 iesoj<»y35) estudy 21 ieitudy21) ettudy 24 <eitudy24) ertudy 37 (estudr37) eitudy 20 ieitudy20) ettudy 09 feitudy09) eitudy 0 (esrudyO) eitudy 12(e«udyl2) ettudy 26 (estudy26) ertudy 06 (eitudy06) estudy 35 (estudy35) ertudy 21(estudy21) Jury 18, 2004 7 Upm July 18. 2004 7 25pn. July 27. 2004 8 2 3pm July 29, 20M 9 20am Jury 29.2004 10 1 lam July 30. 2004 3 04pm July 31. 2004 2 40am July 29. 2004 5 24pm August 1.2004 11 00am August 1.2004 5 32pm August 2. 2004 2 02pm July 20, 2004 11 50am July 26. 2004 1 40pm July 29, 2004 9 23ero July 29. 2004 10 14am Hosted Survey Page: 277 Hosted Survey Round 3 Main Page: 278 Hosted Survey Raw Data: 279 

Cite

Citation Scheme: