Open Collections

UBC Undergraduate Research

Climate science education : retention of concepts and validation of new assessment questions Shinnick-Gordon, Isabel Mar 31, 2014

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
52966-ShinnickGordon_Isabel _ENVR_449_2014.pdf [ 2.45MB ]
Metadata
JSON: 52966-1.0074570.json
JSON-LD: 52966-1.0074570-ld.json
RDF/XML (Pretty): 52966-1.0074570-rdf.xml
RDF/JSON: 52966-1.0074570-rdf.json
Turtle: 52966-1.0074570-turtle.txt
N-Triples: 52966-1.0074570-rdf-ntriples.txt
Original Record: 52966-1.0074570-source.json
Full Text
52966-1.0074570-fulltext.txt
Citation
52966-1.0074570.ris

Full Text

i   CLIMATE SCIENCE EDUCATION: RETENTION OF CONCEPTS AND  VALIDATION OF NEW ASSESSMENT QUESTIONS   by  ISABEL SHINNICK-GORDON     A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF  THE REQUIREMENTS FOR THE DEGREE OF   BACHELOR OF SCIENCE (HONOURS)  in   THE FACULTY OF SCIENCE  (Environmental Sciences)          This thesis conforms to the required standard  ……………………………………… Supervisor   THE UNIVERSITY OF BRITISH COLUMBIA  (Vancouver)  MARCH 2014   ii   ABSTRACT  Misconceptions about climate science are widespread amongst university students. Assessment questions are used to measure student knowledge, knowledge gain and knowledge retention. Seven climate science assessment questions were validated using student interviews, and administered in pre-tests and post-tests to students enrolled in an undergraduate climate science course. Pre-test, post-test and one-year retention tests revealed significant knowledge gains and long-term retention about climate feedbacks and the climate system dynamics. Student interviews revealed common misconceptions about the carbon cycle and system dynamics held by general science students. Results from interviews and assessment administration were used to make recommendations for changes to assessment questions teaching strategies.                   iii  TABLE OF CONTENTS  TITLE PAGE………………………………………………….….…………….………..…....i ABSTRACT…………………………………………………….……………………...……..ii TABLE OF CONTENTS……………………………….…...…………….……….………...iii LIST OF FIGURES………………………………….………..…….……….…….………....v LIST OF TABLES…………………………………………….………………...……….…...vi LIST OF APPENDICES……………………………………………………………………...vi ACKNOWLEDGEMENTS……………………………………..…………………….…......vii INTRODUCTION………………………………………………….…………………………1 PREVIOUS STUDIES  ..……………………………………..................................................2  Misconceptions About Climate Change………………………………………..……..2  Importance of Validating Assessment Questions……………………………………..4  Retention of Scientific Knowledge……………………………………………….…...5 METHODS………………………………………………...……….…………………...…….5  Validation of Assessment Questions……………………………………………….....5 Development of assessment questions ………………………………………..5 Validation interviews………………………………………………………….8 Coding of open-ended test questions………………………………………….9  Assessment Administration………………………………………………………….11   Pre-tests and post-tests………………………………………………………11 Retention surveys…………………………………………………………….11 RESULTS AND DICUSSION………………………………………………………………13 Climate Feedback Concepts………………………………………………………….13 Student interviews: …………………………………………………………..13 Pre-tests and post-tests………………………………………………………15 Long term retention…………………………………………………………..18 CO2 Trajectory Question……………………………………………………………..21 Student interviews……………………………………………………………21 Pre-tests and post-tests………………………………………………………24 Long term retention…………………………………………………………..29 iv  Ranking Carbon Reservoirs and Flows………………………………………………32 Student interviews……………………………………………………………32 Pre-tests and post-tests - ranking reservoirs………………………………...34 Pre-tests and post-tests - ranking inflows …………………………………...37 Pre-tests and post-tests – ranking outflows …………………………………39 Distribution of CO2 in the Atmosphere………………………………………….…..41 Student interviews……………………………………………………………41 Pre-tests and post-tests……………………………………………………....43 SUMMARY AND CONCLUSION…………………………………….………….…..……44 Suggestions for Changes to Assessment Questions………………………………….45 Recommendations for Teaching……………………………………………………..45 REFERENCES CITED……………………………………….……………………………...46 APPENDICES……………………………………………………………………………….49                   v  LIST OF FIGURES   Figure 1. CO2 trajectory problem by Sterman and Booth Sweeney (2002)...…………...……4 Figure 2. Example of amplifying feedback question answer and coding…………….…...…10 Figure 3. Example of a feedback question answer and coding………………...….….…...…10 Figure 4. Pre-test and post-test scores on feedback questions……………………………….16 Figure 5. Amplifying feedback mechanisms used in pre-tests and post-tests...…….……….17 Figure 6. Amplifying feedback mechanisms used in pre-tests and post-tests...…….……….18 Figure 7. Retention of feedback mechanism concepts………………….…………….…...…19 Figure 8. Mechanisms chosen by retention group for feedback questions..………….…...…20 Figure 9. Pre-test and post-test scores on CO2 trajectory questions...……………………….24 Figure 10. Common student answers to the CO2 trajectory questions...…………………….27 Figure 11.Mass balance components on CO2 trajectory questions...………………….…….28 Figure 12. Frequency of common incorrect trajectories CO2 trajectory questions........…….29 Figure 13. Retention of CO2 trajectory concepts.….…....….……..……...…………………30 Figure 14. Mass balance components for post-test and retention test……………………….31 Figure 15. Largest and smallest reservoirs listed on pre-test and post-test….…….……..….35 Figure 16. Largest reservoir on pre-test and post-test by EOSC and non-EOSC students......36 Figure 17. Smallest reservoir on pre-test and post-tests by EOSC and non-EOSC students...36 Figure 18. Largest and smallest inflow on pre-test and post test…………………………….37 Figure 19. Largest inflow on pre-test and post-test by EOSC and non-EOSC students….….38 Figure 20. Smallest inflow on pre-test and post-test by EOSC and non-EOSC students…....39 Figure 21. Largest and smallest outflow on pre-test and post test……..…………..………...39 Figure 22. Largest outflow on pre-test and post-test by EOSC and non-EOSC students…...40 Figure 23. Smallest outflow on pre-test and post-test by EOSC and non-EOSC students….40 Figure 24. CO2 distribution question choices………………..……………..……………..…44     vi  LIST OF TABLES  Table 1. Common climate change misconceptions and assessment questions …..……….......6 Table 2. Learning goals associated with assessment questions….………………..………......7 Table 3. Coding scheme for CO2 trajectory question……………………………..………......9 Table 4. Coding scheme for stabilizing feedback question ……..………………..………....10 Table 5. Summary of knowledge retention on feedback questions………………..………...19 Table 6. Comparison of EOSC 340 student scores to literature on CO2 trajectory question .25 Table 7. Knowledge gain from pre-test to post-test and post-test to retention test..………...30   LIST OF APPENDICES  Appendix 1. Assessment questions…..……………………………………………..……......49 Appendix 2. Student answers to of feedback question ….………………..………................57           vii  ACKNOWLEDGEMENTS   I would like to take the time to thank all who made this study possible. I acknowledge the Carl Wieman Science Education Initiative (CWSEI) for assistance with the funding of retention tests and validation interviews.  I would like to thank all of the students who participated in retention tests and in validation interviews, as well as those who took the EOSC 340 pre-tests and post-tests in the past two years. I appreciate the dedication of your time to improve Earth and Ocean Science education at UBC.  Thank you to Megan Barker, Robin Young, and Trish Schulte for allowing and facilitating my solicitation to the UBC BIOL 200 students for validation interviews. I would like to thank Teresa Woodley and Alicia Cairns helping me with the innumerable room bookings I made to conduct retention testing and student interviews.   I would like to thank Dr. Mary Lou Bevier for facilitating the environmental science thesis course, and for all your advice about research topics, time management, writing style and formatting.  Finally, thank you to Dr. Sara Harris, for your encouragement and advice at every single step of this process. Thank you for meeting with me nearly every week of the school year, for all of your feedback and guidance. I would have been lost so many times along the way without your guidance.  1  INTRODUCTION Humans have altered the global carbon cycle in a short span of geologic time, by changing land use and the burning of fossil fuels (Riebeek, 2011). This has resulted in an accumulation of greenhouse gases in the atmosphere, which has caused changes in the global climate (Intergovernmental Panel on Climate Change (IPCC), 2007). The consequences of our actions are already directly affecting natural processes and human well-being (IPCC, 2007). National and international policies are necessary to regulate and change the role individuals, industries and countries play in affecting global climate change. In democracies, where the public plays an important role in policy-making, it is vital that the public has a basic understanding of carbon cycle – and in particular their own role in it, when making decisions about policy that can impact global climate change. Students graduating with a bachelor’s degree in sciences, whether or not they ever studied climate change or carbon cycling in their coursework, should be able to apply learned scientific understanding and tools to shape their political and personal choices which have effects on the Earth’s climate. However, misconceptions about climate science and the carbon cycle are ubiquitous among secondary and university students (Shepardson et al., 2002; Cronin et al., 2009; Niebert and Gropengiesser, 2011). These misconceptions decrease students’ abilities to make informed decisions about policies and practices affecting climate change. Climate science education should address and correct these misconceptions to improve every-day thinking about the carbon cycle and climate change.  Thus, there is a need to measure the long-term (months to years) retention of knowledge after a climate science course, to determine the effect of education on changing every-day thinking.   The effectiveness of climate science education on changing every-day thinking is measured using assessment questions before and after a climate science course. The quality of assessment questions can be improved by validating them using student interviews (Adams and Wieman, 2011). Asking students to think-aloud as they answer assessment questions shows how the students interpret the question (Ericsson and Simon, 1998). Think-aloud interviews also reveal students’ justification and rationale for their answers, revealing knowledge and misconceptions that aren’t captured by their written answers alone (Adams 2  and Wieman, 2011). Thus, student interview can be used to improve assessment questions by adjusting them until their intention created by the instructor is clear to the student. The purpose of this study is to develop assessment questions and measure the retention climate science concepts by university students one year after they complete a university climate science course. Five questions were used to guide the focus of this study: 1. Do the changes to existing questions make them more easy or difficult for students?  2. Do the assessment questions measure the knowledge differential between students who do and do not understand the concepts? 3. What knowledge is gained during an undergraduate climate science course? 4. How much information is retained one year after undergraduate students complete a climate science course? 5. What types of information are retained? Questions one and two are addressed by conducting student interviews to validate assessment questions, and administering the validated questions in the pre-tests and post-tests of an undergraduate climate science course, EOSC 340-Global Climate Change, at the University of British Columbia (UBC). Question three is addressed by comparing pre-test and post-test answers to the assessment questions of students enrolled in EOSC 340. Questions four and five are addressed by comparing the pre-test to post-test knowledge gains with test scores of students who took the post-test again, one year after completing the course. PREVIOUS STUDIES Misconceptions and Climate Science Education Surveys by Bostrom et al. (1994) and Reynolds et al. (2010) demonstrated that in the past 20 years, public awareness of climate change has increased, but misunderstandings about the climate system persist. Common misconceptions include the equating CO2 emissions with general pollution, that greenhouse gases are trapped below the ozone layer, and that the hole in the ozone layer is caused by CO2. Because scientific understanding of climate change has changed so much in a recent time frame, members of the public may not have encountered it in their secondary or post-secondary education. Many current teachers in 3  secondary schools may hold, and pass on their own misconceptions to their students (McCaffrey and Buhr, 2008, Porter et al., 2012). Recent studies have demonstrated that even current secondary, undergraduate and graduate students also have major misconceptions about atmospheric CO2 and the carbon cycle (e.g. Cronin et al., 2009; Shepardson et al., 2010; Niebert and Gropengiesser 2011; Sterman, 2008). In addition to the misconceptions found by Bostrom et al. (1994) and Reynolds et al. (2010), misconceptions among students include that CO2 traps in heat (Niebert and Gropengeisser, 2011, Shepardson et al., 2010), that greenhouse gases like CO2 are generated only by anthropogenic processes (Niebert and Gropengeisser, 2011), and that greenhouse gas emissions must be very large to cause any change to the global climate (Gowda et al., 1998). Fewer studies have investigated misconceptions about specific processes in the carbon cycle, a study on the biological component of the carbon cycle found that many students incorrectly believe that plants gain their mass from carbon or water in the soil, and not from atmospheric CO2 (Hartley et al., 2011). This suggests that there may be other, yet unknown student misconceptions about the specific links in the carbon cycle and processes in the climate system. Understanding climate change requires a good understanding of system dynamics. For example, students must understand the mechanisms that allow carbon to flow to and from the atmosphere, and how these inflows and outflows relate the accumulation of the carbon stock in the atmosphere, which in turn determines the magnitude of global warming (Sterman and Booth Sweeney, 2002). Sterman and Booth Sweeney (2002, 2007) and Sterman (2008) found that undergraduate and graduate students struggled to describe the relationships between inflows, outflows and the stock of carbon in the atmosphere. When given a trajectory of accumulation, students often draw inflow and outflow trajectories that would violate the conservation of mass (Figure 1). Students also draw outflow trajectories suggesting that they believe that natural outflow processes will continue to increase through 2100 to balance increased human emissions, but recent research has demonstrated that most natural sinks are likely reaching their capacity to take up anthropogenic CO2 and so won’t likely increase in the next few decades (Sterman, 2008). Pala and Vennix (2005) found that after taking an introductory system dynamics course the percentage of trajectories drawn that adhered to mass balance principles increased from 71% to 75%. Dutt and Gonzales (2012) 4  found that after a short lesson, the percentage of trajectories drawn adhering to mass balance increased from 20% to 40%.         Figure 1. The carbon emissions task developed by Sterman and Booth Sweeney (2002), with examples of both “correct” and “typical” responses found by Sterman (2008). Adapted from Sterman (2008). Importance of Validating Assessment Questions When trying to determine the source of the misconceptions about the carbon emissions projection task (Figure 1), Pala and Vennix (2005) said that student interviews would have helped illuminate the thought processes behind these misconceptions, but they did not conduct student interviews. The validation of assessment questions using student interviews demonstrates where misconceptions about the subject arise, and how students’ misinterpretation of an assessment question may cause them to answer in a way that creates an apparent misconception about the subject. The use of student interviews to validate assessment questions is an integral part of a standard process of assessment question validation (Adams and Wieman, 2011). However, student interviews are a time-intensive process, and so have not always been used for validating assessment questions in previous studies (for example, Pala and Vennix, 2005). Other methods of validating assessment questions often include review and comment by experts in the field (Rodriguez, 2002, Porter et al 2012). Although experts may be able to choose questions that might assess important aspects of the field, student interviews determine if students are interpreting these questions as asking intended by the question developer (Adams and Wieman, 2011).  5  Retention of Scientific Knowledge Many studies have measured university students’ long term retention, or the knowledge retained after several months to years after learning the material, of many science concepts (e.g. psychology (Rickard et al., 1988), biology (Kastrinos, 1965), general science (Donovan et al., 1969), anatomy (Blunt and Blizard, 1975, Dubois et al., 1969)). These studies have found that one year after completing a course, students retained about two-thirds to three-quarters of what they learned (Custers, 2008). In some cases, retention tests showed no knowledge loss, or even a net gain during the retention interval, a phenomenon often attributed to students encountering similar concepts during courses taken during the retention interval (Rodriguez et al., 2002).  Many previous retention studies focused on factual information retention, using multiple choice and true-false questions (Custers, 2008), while this study measures retention using open-ended written questions. Also, Few retentions studies have tested students’ retention of earth science concepts, which are often more conceptual than factual in nature. Pala and Vennix (2005) and Dutt and Gonzales (2012) measured knowledge gain about climate science concepts, using open-ended written questions, immediately after the lesson or activity, but neither tested long-term (weeks to years) retention of knowledge. Porter et al. (2012) used multiple choice questions to measure retention of climate science concepts in secondary students, and found that students retained most of what they learned five weeks after a lesson. Thus, there is no known baseline of expected one-year knowledge retention of system dynamics and climate science concepts, using open-ended written questions, amongst undergraduate students.  METHODS Validation of Assessment Questions Development of assessment questions  Assessment questions developed and validated in this study are intended to address some of the common misconceptions students have about climate science, and in particular about the carbon cycle. Assessment questions come from 1) current literature about climate 6  science education (Sterman and Booth Sweeney, 2002), 2) assessment questions used in EOSC 340 (S. Harris, personal communication, 2013), and 3) new questions developed to test misconceptions found in previous studies or during student interviews conducted in this study (Table 1). The questions were also matched to learning goals appropriate for a university-level climate science course (Table 2) Table 1. Student and public misconceptions about climate science, and the assessment questions that address these misconceptions. See Appendix 1 for assessment questions. Misconception Assessment question A stock can stabilize even if inflows and outflows are unequal (violation of conservation of mass) (Sterman and Booth Sweeney 2002; Sterman, 2008; Dutt and Gonzales, 2012). 1 Trajectories of flows to and from a stock are likely proportional to the trajectory of the stock (correlational heuristic) (Sterman and Booth Sweeney 2002; Sterman, 2008; Dutt and Gonzales, 2012). 1 The magnitude of flows to and from a stock is proportional to the amount in the stock (student interviews) 4,5,6 Concentration in a reservoir and reservoir size are proportional (student interviews) 4 Familiarity with a reservoir or flow and its size are proportional (student interviews) 4,5,6 When plants respire or decompose, the carbon goes into the soil, not into the air (Ebert-May et al., 2003; Hartely et al., 2011) 5  The net anthropogenic inputs of greenhouse gases must be very large to have an effect on the climate (Gowda et al., 1997) 5 Greenhouse gases are trapped by the ozone layer (Bostrom et al. (1994) and Reynolds et al. (2010)) 7 The concentration and distribution of carbon dioxide is proportional to localized air pollution such as smog (Bostrom et al. (1994); Bord et al., (1998) and Reynolds et al. (2010)) 7 Greenhouse gases are a layer in the atmosphere that trap in heat (Niebert and Gropengeisser, 2011, Shepardson et al., 2010) 7 Greenhouse gases like CO2 are only generated by anthropogenic processes (Niebert and Gropengeisser, 2011) 7 (option 2) CO2 causes a hole in the ozone layer. (Niebert and Gropengeisser, 2011) 7 (option 6)        7  Table 2. Learning goals addressed by each of the assessment questions. See Appendix 2 for assessment questions. Learning Goal Assessment question Predict flow trajectories that conserve mass  1 Predict flow trajectories that are consistent with the limits of natural carbon stocks and flows.  1 Predict likely input and output trajectories for “business as usual”, or stabilized accumulation goals set by the IPCC 1 Predict what happens to stocks and flows when a system is perturbed 1, 2, 3 Generate feedback loops by connecting a logical set of processes 2, 3 Generate processes which can change the global temperature 2, 3 Rank the sizes of earth’s carbon reservoirs 4 Rank the magnitudes of carbon flows to and from the atmosphere 5, 6 Identify the distribution of greenhouse gases in the atmosphere 7 Question 1 is based on Sterman and Booth Sweeney’s (2002) carbon emissions task, and hereafter is referred to as the CO2 trajectory question. The un-validated CO2 trajectory question (CU) was used in EOSC 340 assessments prior to the Fall 2013 post-test, and provides the outflow of carbon from the atmosphere as a line from 1900 to 2010 (Appendix 1, Question 1A). Two validated versions were developed, CO2-trajectory-line (CVL) presents outflow as a line as in CU, and CO2-trajectory-dot (CVD) presents outflow as a single point for the year 2010 only (Appendix 1). The wording of the question instructions was also changed based on interviews. CVD is more similar than CVL to the question developed by Sterman and Booth Sweeney (2002) (Figure 1). To test the difference in student interpretation and answers, both versions were used in student interviews, and the EOSC 340 pre-test and post-test. Questions 2 and 3 (Appendix 1) ask students to generate a stabilizing and amplifying feedback loop for global temperature. The question is adapted from EOSC 340 Fall 2012 post-test (S. Harris, personal communication, 2013). Results from the post-tests indicate that students frequently create loops that are missing steps, or connect “steps” that are not sequentially related. The interview process was aimed at determining whether these mistakes were due to students’ lack of necessary knowledge, or misinterpretation of the question.  Validation interviews were used to develop two versions of the feedback questions: Feedback-validated-no extra arrows (FVN) provides the same number of arrows and boxes as 8  the original question, while Feedback – validated – extra arrows (FVA) provides extra boxes, arrows and connecting words “which causes” to guide the students answer. Questions 4, 5 and 6 were created to determine students’ knowledge of the spatial and temporal scales of the global carbon cycle, by asking them to rank the size of carbon reservoirs and magnitude of carbon flows to and from the atmosphere (Appendix 1). The reservoirs and flows in the questions were chosen using carbon cycle diagrams (National Aeronautics and Space Administration (NASA), 2011) though the language in the question was changed during the validation process to make the questions more clear.  Question 7 aims at assessing misconceptions about the distribution of greenhouse gases in earth’s atmosphere (Table 1). The options given for the distribution of greenhouse gases were 1) trapped below the ozone layer, 2) accumulating above point sources at high density population centers, 3) accumulating and being trapped close to the surface, 4) a gradient from concentrated at the surface and decreasing with altitude (the correct answer), 5) a gradient opposite to the gradient in 4), and 6) accumulation at the poles. (Appendix 1). Validation interviews The assessment questions were validated using think-aloud interviews with sixteen students (Adams and Wieman, 2011). In a think-aloud interview, the student discusses their thought process as they answer each assessment question. Results from think-aloud interviews can be used to determine if students are interpreting the question as intended by the instructor (Adams and Wieman, 2011). Participants in think-aloud interviews were students enrolled in BIOL 200 – Fundamentals of Cell Biology, at UBC in the 2013 Fall term. Participants had not previously taken a university climate science course. Students were solicited for their participation through an email sent to all students enrolled in BIOL 200. Students without a university-level climate science background were targeted because the validation of the assessment questions was intended to make them understandable to general science students – including those who have not yet encountered the content of the questions in their university coursework. Students enrolled in BIOL 200 were targeted because the course enrolment represented a large number of mostly second year science students who likely had not taken a 9  climate science class. Interviews lasted 30-60 minutes, and each student interviewed was compensated with $10 cash. Results from the student interviews were used to make adjustments to the assessment questions. Appendix 1 shows both the first (un-validated) and final (validated) versions of the assessment questions, although several intermediate versions were used during the interviews. The validated versions were administered to the EOSC 340 Fall 2013 class in their end-of-term post-test and to the Winter 2014 class in their start-of-term pre-test. Coding of open-ended test questions Answers to the EOSC 340 Fall 2013 pre-test CO2 trajectory question were coded by the EOSC 340 Teaching Assistant, Joanne Breckenridge, and all other questions were coded by the author. The inter-rater difference between the two individual coders was 11.4% of 158 codes, using the two codes for the CO2 trajectory questions shown in Table 3. Answers to the CO2 trajectory question were coded for adherence to the laws of mass balance (Table 3). Figure 1 shows a correct emissions trajectory, which would receive two marks, although other possible trajectories exist that would receive both marks shown in Table 3.  Figure 1 also shows an incorrect trajectory, which receives one mark, because inflow doesn’t equal outflow at 2100. Other common answers are shown in Figure 10, in the results section.  Table 3. The coding scheme for the CO2 trajectory question. Code Marks Example inflow line was above the outflow line where the accumulation (stock) was increasing +1 Figure 1 the inflow and outflow trajectories meet by 2100 where the stock is stabilized +1 Figure 1 Total 0 to 2  Each of the feedback loop questions were coded for four components (Table 4). The codes were used to give each student a score between -1 and 3 for each question, and the score from each question was added to give each student a total score between -2 and 6.    10  Table 4. Coding scheme for stabilizing feedback question. For the amplifying feedback question, the “loop leads back to decreasing T” is “loop leads to counteract decreasing T” Code marks Example (has characteristic) Example (does not have characteristic All sequential steps are actually related +1 Appendix 2, example 1 Appendix 2, example 2 Loop leads back to decreasing T +1 Appendix 2, example 1  Appendix 2, example 2  All steps involve internally consistent timescales +1 Appendix 2, example 1 Appendix 2, example 3 an incorrect statement -1 Appendix 2, example 3 Appendix 2, example 1 Total -1 to 3   Figure 2 shows an example of an amplifying feedback loop answer receiving maximum marks and Figure 3 shows an example of a stabilizing feedback loop answer receiving minimum marks. Additional examples can be found in Appendix 2.  Figure 2. A typical answer to the amplifying feedback question using ice albedo as a feedback mechanism, and the coding used to assess the question     Figure 3. A typical incoherent feedback loop, which clearly demonstrates the student doesn’t have the knowledge about feedback mechanism needed to answer the question.    Code marks All sequential steps are actually related +1 Loop leads back to decreasing T +1 All steps involve internally consistent timescales +1 an incorrect statement 0 Total 3 Code marks All sequential steps are actually related 0 Loop leads back to counteract decreasing T 0 All steps involve internally consistent timescales 0 an incorrect statement -1 Total -1 11  Assessment Administration Retention of knowledge was measured by comparing matched student scores on pre-tests, post-tests and retention assessments administered to UBC undergraduate students enrolled in the course EOSC 340 – Global Climate Change in the fall term of 2012. Pre-tests and post-tests Pre-tests were administered by the instructor to the students in the course during the first week of classes in Fall 2012, Fall 2013 and Spring 2014. Post-tests were administered by the instructor on the second-to-last day of classes in Fall 2012 and Fall 2013. Neither of these tests was graded for marks, but students earned a small amount of extra credit for participating. Both of these tests were administered in class, and students were not warned in advance, nor expected to have studied beforehand for either test (S. Harris, personal communications, 2013). It was assumed that any incoming group of students to EOSC 340 will share the same general background knowledge, although the retention group may not have the same distribution of student backgrounds as the average population. Retention surveys Students participated in retention surveys under two different circumstances. The first group was those who responded to an email solicitation sent to all students on the Fall 2012 EOSC 340 class list, between mid-October and mid-November, 2013. The students were asked to participate in a “one-year-later survey”, and were offered the incentive of a chance to win one $50 UBC bookstore gift certificate. Students were not told about the nature of the survey, with the intention that the students would not study the material before taking the survey. This meant that their answers on the survey would represent their “every-day thinking” about the carbon cycle concepts tested. The retention surveys were administered in person on UBC campus. Students were given 30 minutes to complete the survey, although most students finished the survey earlier. Of 184 students enrolled in the course, 150 took the post test and were contacted to ask for their participation, and 12 students participated in the retention survey The second group of students were also enrolled in the Fall 2012 EOSC 340 class, and took the retention test as a pre-test for the advanced climate science course, EOSC 442, in the first week of classes in January, 2014. In the EOSC 442 pre-test, two versions of 12  the test were administered as part of the validation of new assessment questions, so only those who answered the feedback question FVN and the CO2 trajectory question CVL were included in the retention group, because these versions of the questions were most similar to the un-validated questions used in all other pre-tests and post-tests. The two groups who took the retention test were put into one “retention group” for this study. It was assumed that the maximum two months between testing times and the possible effects of different testing circumstances would have insignificant effects on the scores of the two groups. In EOSC 340 pre-tests, significant differences were found in the answers to carbon reservoir and flow ranking questions, between students enrolled in an Earth, Ocean or Atmospheric Science program (Geological Engineering, Atmospheric Science or general Earth and Ocean Science), and non-earth science majors. The fall 2012 EOSC 340 class consisted of 18% earth science majors. By combining the retention groups, the proportion is brought closer to the class average, although still has more earth science majors than the class average. The retention group for the feedback question had 16 students, of which 3 were earth science majors (19%), and the retention group for the CO2 trajectory question had 19 students, of which 6 were earth science majors (32%).  Students in the retention group performed slightly better than the class average, with a mean grade of 78%, compared to a 76% class average. However, the a t-test reveals difference is not significant (p=0.39), so students in the retention group may be an appropriate representation of the EOSC 340 class average.  Students in the retention groups received both amplifying and stabilizing feedback questions on the pre-test, post-test and retention test, but received the CO2 trajectory question on the post-test and retention test only. Thus, the Fall 2013 EOSC 340 class answers to the CO2 trajectory question, using matched individuals, were used to estimate the average pre-test to post-test knowledge gain for the CO2 trajectory question. The retention of the climate science concepts was measured by calculating the knowledge gained or lost between the pre-test, post-test and retention survey. The scores between tests were compared using the difference of means, Welch’s t-test to test for significance and Hedge’s g to measure effect size. Effect size measures the change in the score over the pre-post, post-retention or pre-retention interval as the number of standard deviations between the two means, and so shows 13  how size of the knowledge gain or loss (Maher et al., 2013). The retention test results are judged to be the every-day thinking of the students, one year after taking the course, because one year is long after the period of rapid knowledge loss that often occurs immediately after a student completes a course (Custers, 2008). Thus, if students performed better on the post-test than on the pre-test, this would show knowledge gained from the course. If the students performed better on the retention test than on the pretest, than the knowledge gained during the EOSC 340 course improved their everyday thinking about these climate science topics. RESULTS AND DICUSSION Climate Feedback Concepts Student interviews: Student feedback from validation interviews was used to change the format of the feedback question from the un-validated feedback question (FUN) to Feedback-validated-extra-arrows (FVA) (Appendix 1). During interviews, most students answered the FUN not by visualizing the entire loop first, but by writing one of the first things that came to their mind as the effect, and only sometimes tried to make that lead back to affecting global temperature. For example, student 075 wrote a chain of events that were caused by colder global temperatures, but wasn’t able to link the events back to affecting global temperatures (and thus, failed to draw a feedback loop):  “Climate change … it could then affect like weather patterns around the world? … even longer and colder winters, maybe snow instead of just rain in places, that could impact like crop and agriculture, because farmers have adapted like their growing season and things like that to the weather – to the climate they’re used to. But if it gets colder, it means that they would have a shorter growing season, I guess there could be, a lack of food? Would it make sense if maybe, food scientists would have to come up with more genetically modified foods? I’m not sure where this is going…” The extra arrows, and the linking words “which causes” provided between the first boxes were added to the FUN, to create FVA. These changes were supposed to indicate to students that they needed to connect the first box and arrow to the last arrow, which pointed 14  to either cooling or stable temperature, for the amplifying or stabilizing feedback questions respectively. The changes helped some students connect their sequence of events to further changes in temperature. For example, student 84, found the linking words helpful for creating a loop, even though they couldn’t think of the specific mechanism that linked ice retention at the poles to further cooling of the temperature:  Which causes the ice to … the North and South Pole to retain their ice mass? Well not like … will not melt, I guess, which will cause, cause colder … The extra arrows didn’t help all students connect all steps as a loop. For example, student 80 still struggled to complete a loop from the chain of events they drew that were caused by colder temperature. Student 80 knew they had to connect the loop, but they had to add steps unrelated to the first half of their loop in order to return to stable temperatures. “I don’t know how these things would relate to getting colder … I’m trying to relate how, say … like a decrease in species diversity would contribute to colder temperature.  I want to say that for it to get warmer again, some like random thing just has to happen, for it to kind of like, recycle back to the stable temperature. I’m just not sure how to link that into, like the causation of everything. So I want to say somewhere, that maybe the planet moves closer to the sun for some reason … and so that sort of warms it back up.” Student 80 acknowledged that they knew the steps were not connected. This information indicates that in assessments, the “extra steps” or “illogically connected steps” in students loops may not indicate that the student incorrectly thinks two processes are actually connected, but instead may sometimes show that the student could not think of the “missing step” that would close the loop, or may have thought it appropriate to include in their loop processes which “happen simultaneously” but are not connected or caused by other processes in the loop.  Several of the interviewed students who struggled to imagine a stabilizing feedback mechanism used humans as the mechanism to stabilize the temperature. This choice is also common in EOSC 340 pre-tests and post-tests, but the interviews demonstrate that the reasoning and justification behind these answers varies. For example, student 78 used 15  humans as a mechanism because they thought that the first amplifying feedback question asked about a problem scenario, and so the second question asked about how humans could fix that problem. “Oh I see, so we have instead of it, the world being doomed, and becoming colder and colder, we want to find a solution, because the global temperatures are still getting colder,... so I’d probably relate this question to what I was talking about in the last question cause I they’re essentially the same thing, except now we want to return back to stable temperature.” Few of the interviewed students demonstrated understanding that human greenhouse gas emissions lead to warming the global temperature, a misunderstanding that is less clear when only the written answer is assessed. For example, student 74 used human processes as a stabilizing feedback mechanism, but their description of their feedback loop showed they had the misconception that all pollution caused by humans causes global warming. “We are trying to get the global temp back to the stable value; do you have to consider pollution or something? Like in the … somehow we magically get the temp back to stable? Or like, we’re doing the same, while we’re making pollutions, do we have to consider that part?” Pre-tests and post-tests Because of the mixed results among students interviewed, both FVN (feedbacks – validated – no extra arrows) and FVA (extra arrows) were administered in the EOSC 340 pre-test and post-test. Figure 4 shows the difference between FVN and FVA is insignificant in both the pre-test or post test (p = 0.26 and 0.53). Scores are slightly lower for FVN in the pre-test and slightly higher in the post-test. This suggests the extra boxes and arrows provided in FVA may help students who are not familiar the concept of feedbacks but that the extra arrows do not help, and may even hinder students who are more familiar with feedback concepts (post-test). 16   Figure 4. Pre-test and post-test scores (scores range from -2 to 6) for the feedback questions, for two versions of the question. Means and standard errors are plotted. FVN = feedbacks – validated – no extra arrows. FVA = feedbacks-validated-extra arrows As in the student interviews, some students in the pre-test did not answer the question as a loop, and instead drew only the causes, and the effects of colder global temperature. Students were much more likely draw these un-closed loops if they received FVA rather than FVN: 10% of FVN and 32% of FVA amplifying feedback answers had incomplete loops, and 2% of FVN and 25% of FVA of stabilizing feedback had incomplete loops. FVA provides an extra arrow before and an extra box and arrow after “temperature gets colder.” This may encourage students to only fill in what appears to be the two or three missing pieces: one cause of colder temperature and two effects This phenomenon was insignificant in the post-test, and was not coded for.  In the pre-tests and post-tests, for the amplifying feedback, the question version did not affect the types of mechanisms chosen for the feedback loops by students (Figure 5). Both groups also showed knowledge gain from pre-test to post test. For example, the frequency of answers using mechanisms categorized as “other,” which were never correct (see Appendix 2), decreased from pre-test to post-test (Figure 5, Figure 6).  17   Figure 5. Frequency of feedback mechanisms listed on the pre-test and post-test for the amplifying feedback question.  As in the answers to the amplifying feedback question, there was also a decrease in choosing “other” as a feedback mechanism for the stabilizing feedback question from pre-test to post-test. The variety of correct feedback mechanisms chosen is much higher in the post-test than in the pre-test, showing knowledge gained about a variety of stabilizing feedback mechanisms (Figure 6). For example, the frequency silicate weathering, Planck, and clouds as feedback mechanisms were chosen as mechanisms was much higher in the post-test than in the pre-test. However, although students were taught about lapse rate as a feedback mechanism during the course, the frequency of choosing this mechanism is not high in the post-test. The mechanism “plants dying”, on the other hand, was not mentioned as a feedback mechanism during the course, yet remains a persistent choice in both pre-test and post-test. While the absence of lapse rate as a choice doesn’t show students didn’t learn about this mechanism during the course, the persistence of plants dying and humans shows that many students still have some misconceptions about climate feedback loops 18   Figure 6. Frequency of different feedback mechanisms used in pre-test and post-test for the stabilizing feedback question. Long term retention Like the larger student groups, students in the retention group also showed knowledge gain from pre-test to the post-test, as well as the retention of some knowledge one year after completing the EOSC 340 course (Figure 7, Table 5). Both knowledge gain from pre-test to post-test and knowledge loss from post-test to retention test were significant with p<0.05 and effect sizes of 1.50 and 0.99 respectively. The knowledge gain from pre-test to retention test had an effect size of 0.66, although the average retention and pre-test scores were not statistically different (p=0.09).  Effect sizes greater than 0.6 are considered to be large changes of knowledge (Maher et al., 2013), and effect sizes of the magnitudes in this study are comparable to previous studies comparing pre-test and post-test knowledge (Gottesman and Hoskins, 2013). Thus, one year after completing a climate science course (EOSC 340), students still demonstrate knowledge about feedback loops gained during the course. 19   Figure 7. Scores on feedback questions showing knowledge gain and loss from pre to retention test.  Scores range from -2 to 6. The retention group is a subset of the Fall 2012 class.  Table 5. Comparison of test scores for retention students throughout the test sequence (n=16). Means are +/- standard error. A * indicates a p-value showing a significant difference between tests. Effect size is measured using Hedge’s g (Maher et al., 2013).  Pre-test to post-test Post-test to retention test Pre-test to retention test Mean score test 1 2.37 +/- 0.55 (pre) 5.06 +/- 0.32 (post) 2.37 +/- 0.55 (pre) Mean score test 2 5.06 +/- 0.32 (post) 3.67 +/- 0.45 (retention) 3.67 +/- 0.45 (retention) Mean change 2.69 -1.39 +1.30 Welch’s t-test: significance p-value 2.0 x 10-4* 0.015 * 0.09 Effect size (Hedge’s g) Small/med/large/very large (Maher 2013) 1.50 Very large -0.99 large 0.66 large The knowledge loss from post to retention test in each of the categories coded (see Table 3) was less than the knowledge gain from pre-test to post-test, so there was no particular component of feedback loop concepts in which students were more likely to revert back to their pre-course thinking. One exception is that there was almost no knowledge lost in students’ ability to generate mechanisms which resulted in the appropriate change in temperature (getting colder or stabilizing). In all tests, the most common correct amplifying feedback mechanism used was ice-albedo: 1/3 of students listed this mechanism in the pre-tests, all listed it in the post test, and 2/3 listed it in the retention test (Figure 8). Other mechanisms listed by three or fewer retention students were water vapor and ocean uptake – which students used in the pre and retention test, but not the post test. “Other” mechanisms 20  listed by students were always incorrect, and were used by 39% of students in the pre-test, 9% in the post-test and 18% in the retention test. The pre-test to retention test decrease in the use of “other” mechanisms shows a net knowledge gain of information about climate feedback mechanisms (Figure 8).  Figure 8. Mechanisms chosen by retention group students (n=16) in the pre-test, post-test and retention test, for the amplifying and stabilizing loops.  In the stabilizing feedback question, no retention students listed silicate weathering or clouds as feedback mechanisms during the pre-test, but one to three students listed one of these on the post-test and retention test, showing that students learned these new mechanisms, and remembered them even a year after completing the course (Figure 8). Planck and lapse rate as mechanism choices appear on the post-test, but not on the retention test, showing that these mechanisms may be forgotten one year after the course. While choosing humans as a mechanism was common in the pre-test, only two of sixteen used this in the pre-test and post-test, showing knowledge was gained and retained about other, natural feedback mechanisms.    21  CO2 Trajectory Question Student interviews Validation interviews demonstrated that even science students with no earth science background were often able draw trajectories containing the components of mass balance to code the question: that inflow is more than outflow when stock is rising, and that inflow=outflow when stock is stable (Table 4). These students did not rely on understanding of climate systems, but instead cited university level math, biology, first-year science courses or “intuition” as their reasoning for their answers. When asked follow up questions, many of the students struggled to generate examples of natural outflows of carbon from the atmosphere, but most were able to generate examples of inflow, including humans “burning wood” or “burning fuels”. For example, student 89 used knowledge about mass balances to draw trajectories with three of four mass balance components, but also acknowledged they didn’t know much about the actual processes that could lead to the trajectories they drew:  “If we’re uh, reaching stabilization by the year 2100, that the outflow would have to equal the inflow, because if it’s like, stuff leaving, versus stuff coming in its being stabilized at like a steady level then I suppose it would have to - they would have to meet up at some point … I don’t quite understand climate change as well as … so when I think specific processes I think like things that would increase carbon emissions or decrease carbon emissions.” Ten of the sixteen students interviewed changed their answer at least once after attempting to answer four written follow-up questions, asking the student to describe the human and natural processes that could account for the trajectories that they drew (Appendix 1). Some of these students changed their trajectory so that outflow did not increase by as much, and others who didn’t change their answer explained that their trajectory was unrealistic because the outflow trajectory they drew increased more than natural processes would realistically allow. Student 77 explained this, during their third attempt drawing the trajectories, adjusting both inflow and outflow to increase at a slower rate:  “I guess that’s rather idealistic to assume that the natural processes will just catch up to human lines, so maybe it’s more somewhere in the middle”.  22  This demonstrates that when students are only asked to draw trajectories, they aren’t necessarily thinking about climate processes, and whether their trajectories are “realistic”, given the capabilities of humans to control their emissions and the capacity of the environment to sequester carbon.  Interviews demonstrated that students did not struggle to define CO2 stock as “the amount of carbon in the atmosphere”, and most were able to define CO2 inflows and outflows. However, some struggled to define the concept of flows or were unable to differentiate between inflow and outflow. For example, when student 76 was asked to define inflows and outflows, they said, “they’re both just kind of the same, because they’re both removing and stabilizing.” This student incorrectly thought both inflow and outflow remove carbon from the atmosphere. Interviews showed some of the reasoning behind common incorrect trajectories drawn on pre-tests and post-tests. A few students in pre-tests and post-tests drew outflow crossing inflow. Six of sixteen students interviewed drew outflow crossing inflow in their first attempt at drawing the trajectories, with vague explanations such as: “In order to stabilize inflow I guess it would go up from that little dot … so through 2100 I guess outflow would have to go up above inflow? I’m guessing, to stabilize the net graph” (Student 77).  However, five of these six changed their answer after reading the follow up questions, realizing that if outflow crossed inflow, the stock would decrease, as described by student 84,  “If that happens, then there will be a decrease from the stock.” Some students even suggested that outflow could never cross inflow. For example: “Not actually because well this is like a rate, right? It’s not like, if we were just talking about stock, there’s no way like you could be- you could be getting rid of all the carbon, I mean more than there is carbon in the atmosphere –but because this is like a rate, a rate per year.” (student 88). “Logically I don’t think it makes sense for an outflow to be higher than an inflow.  You can’t have more going out than came in, if that makes sense.” (student 80). 23  This belief is probably tied to a specific scenario where if the stock is entirely depleted, outflow cannot exceed inflow, as there is nothing left in the stock. However, the figures provided to the students in this question demonstrate that stock (in Gton C) is always far larger than could be removed in a given year, given outflow values (in Gton C/year), so outflow could cross inflow. However, because this question never asks students how the inflows and outflows would behave such that the stock would decrease (in which outflow must exceed inflow), the answers in pre-tests and post-tests do not assess the prevalence of this misconception in the student body. Many students described their inflow, and sometimes their outflow trajectories as “stabilizing” by 2100. Students sometimes correctly associated the word “stabilization” with the mass balance of inflow and outflow, as student 82 described: “I’m thinking, as it stabilizes… it the inflow and outflow should meet.” However, other students said that the stock “stabilized” when inflow “stabilized,” but defined stabilizing as the slope of the inflow line levelling to horizontal, not when inflow met outflow. For example:  “Human carbon emissions would be inflow. I guess inflow of carbon into the environment? I’m guessing that’s what that would mean … so I just drew it stabilizing by 2100” (Student 77). The interviews indicated that the dot version of the question is more confusing to students. No students who received the line version of the question had difficulty identifying that the outflow trajectory given showed the outflow from 1900 to 2010, but students seeing the outflow dot struggled to understand that it represented outflow for a single point, for the year 2010. Student 76 thought the dot meant the answer was asking for another dot: “I can relate the new line into the old line that they gave me, so for the inflow should be a continuation of the lines that … just said inflow, and for outflow I’m not really sure what’s going on, so … I think it’s another point.”  Student 74 thought the dot represented a discrete outflow “event”, where humans found a way to sequester large amounts of carbon in order to reduce the effects of human emissions. 24  “I’m thinking maybe at that time … there’s a sudden event that takes all the carbon away. Because probably at 1900s and 50s, people not really focusing on these stuffs at the moment at least? That makes sense, because after 2000, people start to realize, “oh that’s a huge problem,” and they start to do something, and … it removes carbon dioxide, but before that, we haven’t found a way to take out the CO2, or it’s less significant?” The students given the dot test who drew an outflow trajectory decided to guess that the dot did represent the yearly outflow for the year 2010 (as is stated in the written introduction to the problem). However, student interviews demonstrate that even when students receiving the dot version answer correctly, they do it with more difficulty than students who receive the line version.  Pre-tests and post-tests Figure 9 shows that both versions (CO2 trajectory-validated-dot (CVD) and –line (CVL) (see Appendix 2 for CVD and CVL) show the knowledge gains by students during the course. However, in the pre-test students find CVD much more difficult -test, as scores for CVD in the pre-test are significantly (p<0.05) lower than for CVL.  Figure 9. Mass balance score on CO2 trajectory question for pre and post-test, for both students receiving dot and line versions of the question. CVD = CO2 trajectory question – validated – dot, CVL = CO2 trajectory question – validated – line. 25  The additional information provided in CVL showing all outflow from 1900 to 2010 may have made this version less difficult. The outflow line is the main difference between this question and the question used by Sterman and Booth Sweeney (2002, 2007), and Sterman (2008). EOSC 340 students answering CVL more often answered with correct mass balance, but incorrectly drew outflow increasing substantially, than students in studies by Sterman and Booth Sweeney (2002, 2007) and Sterman (2008) (Table 6). Sterman (2008), Sterman and Booth Sweeney (2002, 2007) describe the trajectories where inflow increases after 2010 and does not decrease as “correlational reasoning” between inflow and the stock of carbon, and one third of EOSC 340 students used this type of reasoning (Table 6). Table 6. Coding of student answers to the CO2 trajectory question, comparing dot and line tests, as well as previous literature (all previous literature uses a “dot” version). If the component was not coded for in a paper, the box is left blank. Sterman and Booth Sweeney (2002) asked the question as a multiple choice question. Sterman and Booth Sweeney (2007), and Dutt and Gonzales allowed for +/- 0.5 between inflow and outflow at 2100. Dutt and Gonzales “description” is the paper question alone while in “experience” the students first learned about mass balance concepts using a computer simulator.  Pre-test CVD Pre-test CVL Post-test CVD Post-test CVL Sterman and Booth Sweeney (2007) Dutt and Gonzales (2012) Sterman and Booth Sweeney (2002) Description Experience Inflow>outflow at 2100 36.9 31 17.8 10.6 63 78 50 - Inflow=outflow at 2100 52.2 59.0 68.9 74.5 31 20 43 - Inflow<outflow at 2100 10.9 10.0 13.3 14.9 6 2 7 - Outflow rises after 2010 95.8 100 95.6 100 72 - - - Inflow rises after 2010 92.7 95.1 84.4 95.7 - - - 47 Inflow falls at some point 12.7 13.1 24.4 17.0 30 - - 39 “correct” trajectory 6.7 8.7 22.3 13.0 - - - 18 Uses correlational reasoning 33 29 24 31 - 82 60 - The two versions of the question were tested because in 2012 post-tests, students frequently drew outflow unrealistically increasing. The capacities of natural sinks will likely cause outflows of carbon from the atmosphere to stay constant or decrease through 2100 (Sterman, 2008). Previous studies providing only the dot for outflow (Sterman and Booth Sweeney 2002; Sterman, 2008; Dutt and Gonzales, 2012) found fewer students drawing dramatically increasing outflow, but also saw a majority of students violating principles of mass balance by assuming a stock can stabilize while inflow exceeds outflow. While the coding of answers was not the same in all studies, there are significant differences between 26  the EOSC 340 pre-test and post-test results and those by Sterman (2008) and Sterman and Booth Sweeney (2002, 2007) (Table 6). Most notably, it appeared that students in both the pre and post-tests of EOSC 340 did not struggle to answer maintaining the conservation of mass, as more than half had inflow=outflow at 2100 in all tests, compared to only 11% in Sterman and Booth Sweeney’s 2007 study. On the other hand, nearly all of the EOSC 340 students drew outflow increasing after 2010, compared to only 70% in Sterman’s 2008 study. EOSC 340 students also drew inflow increasing after 2010 far more frequently, and had inflow decrease far less frequently than the previous studies. Overall, the “correct” trajectory, defined as having inflow=outflow at the end, and inflow having to fall at some point, was found in 18% of answers by Sterman and Booth Sweeney (2002), which is about the same percentage as the average in EOSC 340 post-tests. Sterman and Booth Sweeney (2002) assume during their study that the correct outflow trajectory is flat or decreasing slightly. In written explanations of their trajectories, students showed they knew inflow and outflow had to balance, and that inflow had to decrease a lot to meet outflow (Sterman and Booth Sweeney, 2002). Sterman and Booth Sweeney hypothesize that because they include in the wording of the question that that removal in 2010 is currently half emissions, this tells students that outflow should not increase much, if at all through 2100. However, this may in fact result in the higher frequency of “correlational heuristics” seen in Sterman and Booth Sweeney’s tests, because the wording of the question might be interpreted by students that outflow should always be half of inflow, and so outflow and inflow would remain parallel, but never meet. In our version of the question, we do not explicitly state that the outflow “dot” is half of emissions, but instead say it represents the outflow for the year 2010. This could be why more EOSC 340 students draw trajectories that maintain the conservation of mass than in Sterman and Booth Sweeney’s studies. Figure 10 shows typical trajectories drawn by EOSC 340 students. Figure 10 C-F show trajectories violating mass balance. In EOSC 340 pre-tests less than 10% of either CVD or CVL answered correctly (Figure 10A), and in the post test 22% of CVD and 13% of CVL answered correctly. Therefore, although the dot version of the question appears to be more difficult for students, this question is also more likely to generate the correct response by students. This indicates that given the dot version of the test, students may be more likely to 27  incorporate both their knowledge of mass balance, and of climate processes, in order to provide the correct answer. Another potential cause for some of the differences between the EOSC 340 question and the previous studies is that the previous studies ask students to draw outflow first, while we ask first for inflow. When student are asked to draw inflow first, they may be more likely to use correlational reasoning to draw the inflow matching stock, while if they must draw outflow first, they must think about how outflow and inflow might interact. However, whether students in interviews were asked to draw inflow or outflow first, they always drew inflow first, so the order in the wording of the question does not influence students’ answers in this way.  Figure 10. Common student answers to CVD or CVL. A. “Correct trajectories”. B.  Trajectories with conservation of mass, but unrealistic outflow. C. Inflow and outflow are parallel. D. Inflow and outflow get closer together. E. Inflow and outflow get farther apart. F. Outflow crosses inflow.  Answers marked with * (A-B) received 2/2 marks, and the “incorrect” trajectories above (C-F) received 1/2 marks. A* F E D C B* 28  Figure 11 shows that in both the pre- and post-test, students who receive the dot version of the test less frequently include 3 of the 4 components of a correct mass balance trajectory: inflow>outflow when stock is rising, inflow=outflow at 2100, and inflow close to outflow at 2100. Students receiving the dot version slightly more frequently had the greatest difference at the maximum part of the curve. There was knowledge gain from pre to post test in three of these components, although slightly fewer students in the post test had inflow>outflow when stock is rising. 39% of dot test takers in the pre-test correctly filled in the past trajectory, compared to 86% in the post-test, which means students improved in there was also an increase in students drawing past outflow trajectories adhering to mass balance.   Figure 11. Mass-balance components for CO2 trajectory question. A * marks components used in coding (assigning scores)  Figure 12 shows that in the post-test, fewer students drew the incorrect trajectories of having inflow parallel to outflow or inflow and outflow moving further apart. Slightly more students in the post-test drew outflow crossing inflow than in the pre-test. 29   Figure 12. Common incorrect trajectories drawn by students in the pre-test and post-test. See Figure 10 for examples of these trajectories drawn by students. Long term retention Unlike in the feedback mechanism questions, students showed knowledge gain both from pre-test to post-test and from post-test to retention test using the CO2 trajectory question (Figure 13, Table 7). The Fall 2013 class averages and the retention group overlap scores overlap on the post-test, so the Fall 2013 class average is a reasonable proxy for EOSC 340 student pre-test to post-test knowledge gains, and the retention group serves as an estimate of post-test to retention test retention.  On the post-test, the retention group scored lower than the Fall 2012 class average, which is unexpected because the retention group scored higher than the class average on the pre-test and post-test for the feedback questions. However, the coded CO2 trajectory question and the feedback questions measure different types of knowledge, so it is possible that the retention group was had a greater knowledge of feedback mechanisms and less of system dynamics than the class average.  30   Figure 13. Scores on the mass balance components of the CO2 trajectory question, showing knowledge gain and loss from pre-test to retention test.  Scores range from 0-2. Table 7. Knowledge gains for retention group for CO2 trajectory question. Scores are out of 2. Effect size  is number of standard deviations, and magnitude of effect size is according to Maher, 2013. A * indicates a statistically significant p-value.  Pre-test to post-test (Fall 2013) Post-test to retention test (retention group) Mean score test 1 1.18 +/- 0.09 (pre-test) 1.37 +/- 0.17 (post-test) Mean score test 2 1.61+/-0.09 (post-test) 1.84 +/- 0.086 (retention test) Difference +0.23 +0.47 Welch’s t-test: significance p-value 0.001* 0.02* Effect size 0.70 Large 0.79 large Although only two codes were used to measure knowledge gain in Figure 13, Figure 14 shows that knowledge was gained in three of four coded mass balance components, after the post-test. Only for the component where the difference between inflow and outflow is maximum at 2030, decreased from post-test to retention test, although fewer than 25% of students on both the post-test and retention test included this.  31   Figure  14. Mass balance components for CO2 trajectory question in post-test and retention test (n = 19). Of four common incorrect trajectories drawn (Figure 10 C-F), there was not an increase in the frequency in any of these from the post-test to the retention test, only 1-3 students drew any of the incorrect trajectoreis shown in Figure 10 in either post-test or retention test.  In previous studies, the improvement in scores from post-test to retention test has been attributed to coursework students took in the interim between the post-test and retention test (Rodriguez et al., 2002). For this reason, students who participated in the first round of retention tests were asked if they took any climate science courses in the interim, and 25% of these students said that they had. The proportion of the second group of retention test takers who had taken an additional climate science course is unknown. Additonal coursework (not necessarily related to climate science) may account for some of the improvement from post-test to retention test, because Pala and Vennix (2005) found performance on a similar stock-and-flow problem improved after one semester for both students enrolled in a system dynamics course and a control group of students not enrolled in a system dynamics course. Validation interviews conducted with students without earth science backgrounds, demonstrated that students use knowledge from non-climate science courses to answer this question.  For example, student 79 said when they were answering the question, “mostly when I was thinking of this, I was thinking of math”, and also attributed their ability to 32  interpret the graphs to their second year biology course. These observations support the conclusion that some of the knowledge gain may be due to additional coursework taken by the students during the retention interval, regardless of whether or not that included climate science courses. Ranking Carbon Reservoirs and Flows Student interviews Interviews with students demonstrated that there are three major misconceptions of proportionality about carbon reservoirs and flows: 1) that flow rates in and out of a reservoir are proportional to reservoir size, 2) that the concentration of carbon in a reservoir is proportional to reservoir size and 3) that the size of a reservoir or flow is proportional to one’s familiarity with that reservoir or flow.  In interviews, some students believed that large reservoirs must correlate with large inflows and outflows of carbon, while others said that the absence of this proportionality in the carbon system is a “paradox”. Student 81 said: “So the largest flow would mean, I guess the largest flow would come from the biggest reservoirs, and oceans are also pretty big.”  In the ocean carbon uptake/degassing system, the size of the flow is large because of the large surface area of the ocean. However, the largest reservoir of carbon, rocks, correlates to some of the smallest direct flows of carbon into and out of the atmosphere (volcano output and silicate weathering). Most students acknowledged that weathering rocks was not a significant outflow of carbon from the atmosphere, but they still acknowledged an apparent “paradox” when they said that weathering rocks was a small flow, yet rocks the largest reservoir: “That’s some paradox here. Like I think rocks must compose a lot of carbon right, but I don’t think um … it’s take up from the atmosphere. It must be a very, very slow process like … so … I don’t know, yeah it takes out least, must be the rocks” (student 88) 33  The second misconception of proportionality is that carbon concentrations in a reservoir are proportional the total amount of carbon in a reservoir. Students used this reasoning to rank atmosphere as a small reservoir, and biomass as a large reservoir. “The reason why I want to say atmosphere is the smallest is I think, well the percentages that make up the atmosphere so I think like predominantly nitrogen and stuff like that?” (Student 89).  “And carbon living biomass, yes because all living biomass consist of carbon” (student 85, in ranking biomass as largest reservoir) “We’re made of quite a bit of carbon, that’s why I’m putting a little higher” (student 79, ranking biomass 3rd largest reservoir) The third misconception of proportionality is that the size of a reservoir or magnitude of a flow is proportional to a student’s familiarity with that reservoir or flow. This was often the case with silicate weathering, where students didn’t know anything about it, and so ranked it as last: “I’m not familiar with weathering rocks, so I’m going to put that as least” (student 84). The problem with Question 6 is that in this case the rank order of familiarity of outflows is the same as the order of the magnitude of the outflows. This makes this question a poor assessment tool to understand student’s ability to rank the importance of outflows, because it isn’t able to differentiate between the students who answer correctly because they know the relative amounts of each flow, and those who answer correctly because they know nothing about oceans and less about weathering rocks. Related to the proportionality of familiarity is the belief that the human perturbations on the flows are much more important than the natural processes. Students most often incorrectly rank human-caused fossil fuel emissions as the largest inflow of carbon to the atmosphere. Assaraf and Orion (2005) found students similarly overemphasized the role of humans in the hydrologic cycle. The justification by the interviewed students is sometimes based on familiarity, but more often the justification for ranking human emissions above natural emissions is due to a misinterpretation of the question. Students misinterpret the question by thinking that the question asks to rank the net inflows, rather than the total inflows of each process. Therefore, students justify their answers in ways such as student 75: 34  “Because all of the other ones will like the ocean, volcanoes, and just from respiration, they’re natural processes I guess, that occur already, but the fossil fuel emissions is something that we’re adding to the environment.” Another misconception held by interviewed students is about the role of decomposition and respiration as an input of carbon into the atmosphere. Previous studies demonstrate that students struggle with the roles of photosynthesis and respiration in the carbon cycle (Ebert-May et al., 2003; Hartely et al., 2011). Students 81 and 76 also demonstrated this confusion:  “Respiration by plants and decomposition – I know this one’s the lowest cause they would take away CO2 from the atmosphere, so I’m going to put d here. Cause when they respire they take CO2, H2O and they lose glucose and oxygen whatever, so I said that was the least adding,” (Student 81) “I’m not sure if the respiration by plants actually take carbon dioxide out of the atmosphere, versus photosynthesis.” (Student 76) Pre-tests and post-tests - ranking reservoirs For the reservoir and inflow ranking questions, the question asked students to rank five items, which means there were a total of 120 possible answers. Thus, the number of students who chose the exact correct answer was very low. Therefore, as a proxy for the average rankings chosen by the class, the frequencies of choices listed as largest and smallest were used instead of complete individual student answers.  Figure 15A shows that three times as many students in the post-test than in the pre-test correctly chose rocks are the largest reservoir of carbon. Students were also less likely to incorrectly choose fossil fuels or the atmosphere as the largest reservoir of carbon, and the percentage of students who incorrectly believed biomass represented the largest reservoir decreased from 15% to 0%. Fewer students incorrectly picked rocks, oceans or fossil fuels as the smallest carbon reservoir in the post-test than the pre-test, while more picked atmosphere or biomass – two reservoirs which contain similar and small amounts of carbon. Those who picked atmosphere increased from 27% to 61% (Figure 15B). 35   Figure 15. Comparison of all students’ pre-test and post-test rankings of the largest carbon reservoir (A) and smallest (B). pre-test n = 103 and post-test n = 97. In the pre-test, the ranking for largest and smallest reservoirs was generally evenly distributed across the all reservoirs. Only oceans stood out as most frequently picked as largest and least frequently picked as smallest. This may be because when students aren’t familiar with the reservoirs (in the pre-test) they may to believe the largest carbon reservoir is the one they perceive to have the largest volume. This was the case in the student interviews, where most students interviewed believed oceans contained the largest volume, though some believed rocks, or even atmosphere contained the largest volume. In the pre-test, earth science majors were more likely to pick rocks, and less likely to pick oceans, atmosphere or biomass as the largest reservoir than non-earth science majors. In the post-test, the gaps earth science and non-earth science majors closes, and both groups demonstrate knowledge gained during the course (Figure 16).         A B 36   Figure 16. Reservoir listed as largest in the winter 2014 pre-test (1A) and Fall 2013 post-test (1B) of EOSC 340 by students of different education backgrounds. Reservoirs are listed from largest on the left to smallest on the right. In the pre-test, non-earth science majors most often picked atmosphere as the smallest reservoir, while earth science majors most often picked biomass. In the post-test, more than 60% of both groups picked the second smallest reservoir, atmosphere, and about 20% of both groups correctly chose biomass as the smallest reservoir (Figure 17). As in the choices for the largest reservoir, the gaps between the two groups closed in the post-test, and both groups demonstrated knowledge gain by most often choosing one of the two smallest reservoirs.  Figure 17.  Reservoir listed as smallest in pre-test (A) and post-test (B). pre-test n = 103 and post-test n = 97. Reservoirs are listed from largest on the left to smallest on the right.  B A A B 37  Pre-tests and post-tests - ranking inflows  Before and after the course, more than half of students incorrectly chose fossil fuel emissions as the largest inflow of carbon into the atmosphere (Figure 18). The choice of ocean outgassing as the largest inflow increased from 15% to 25%, while respiration and decomposition, deforestation and volcanoes all remained below 10% in both pre-test and post-test. Student interviews indicated that some students are confused as to whether the question asks for the largest “net inflows” or “total inflows”, which caused some of the interviewed students to choose fossil fuels over decomposition, because the viewed decomposition as a balanced by photosynthesis. The similar pattern in both the pre-test and post-test indicates that this misinterpretation of the question may be more widespread amongst the EOSC 340 class than it was in the group of students interviewed. In the pre-test, respiration is chosen as the largest inflow by fewer than 10% of students, and was the picked most frequently as the smallest inflow in both pre-test and post-test. Interviews indicated some students confuse the processes of respiration and photosynthesis, believing that respiration takes carbon out of the atmosphere, rather than adding carbon to it, and thus rank respiration as smallest because it is actually a “negative” inflow of carbon to the atmosphere. This justification may be why the frequency respiration is chosen as the largest inflow is low in the pre-test.  Figure 18. Carbon inflow listed as largest (A) and smallest (B) by all students on the pre-test and post-test. Inflows are listed from largest on the left to smallest on the right. Figure 19 shows that the increase the pre-test to post-test increase in oceans as the choice for largest reservoir is mostly among earth science majors. Thus, the change may be B A 38  due to other courses these students are taking, as non-earth science majors did not show the same knowledge gains. The misconception that fossil fuels is the largest reservoirs is persistent among both earth science majors and non-earth science majors, and decreased slightly in the post-test for both groups, but only by a small percentage.  Figure 19. Inflow listed as largest by students in the pre-test (A) and post-test (B). Inflows are listed from largest on the left to smallest on the right. Figure 20 demonstrates that the inflow chosen as smallest is not different between students from earth science and non-earth science backgrounds. The exceptions are that in the post-test, respiration is picked by nearly twice as many non-earth science than earth science students as the smallest inflow, while deforestation is picked as the smallest inflow by far fewer non earth science majors.   Figure 20. Inflow listed as smallest by earth and non-earth science majors on pre-test (A) and post-test (B). Inflows are listed from largest on the left to smallest on the right. A B A B 39  Pre-tests and post-tests – ranking outflows  More students chose photosynthesis and fewer chose oceans or weathering rocks as the largest outflow carbon outflow in the post-test than the pre-test (Figure 21). Over 60% of students in both the pre-test and post-test correctly chose weathering rocks as the smallest outflow. Results from student interviews suggest that this trend may be due to an increase in students’ familiarity with the processes rather than an increase in knowledge about them. Thus, what appears to be an increase from pre- to post-test of the misconception that weathering rocks is the largest outflow, may instead show an increased familiarity with the process of weathering rocks.   Figure 21. Outflow listed as largest (A) and smallest (B) on pre-test and post-test. pre-test n = 103 and post-test n = 97. Outflows are listed from largest to smallest from left to right. Figure 22A shows that on the pre-test, 20% more earth-science majors correctly picked photosynthesis as the largest outflow than non-earth science majors. On the post-test,  the gap between the two groups of students closed, with fewer of both earth science and non-earth science majors correctly choosing photosynthesis as the largest outflow (Figure 22B). Earth science majors more frequently pick weathering rocks as the smallest outflow in both pre-tests and post-tests (Figure 23), which may be because earth science students are more likely than non-earth science students to take other courses during the term where they may also learn about processes such as weathering rocks. B A 40   Figure 22. Outflow listed as largest by earth and non-earth science majors on pre-test (A) and post-test (B). Outflows are listed from largest on the left to smallest on the right.  Figure 23. Outflow listed as smallest by earth and non-earth science majors on pre-test (A) and post-test (B). Outflows are listed from largest on the left to smallest on the right. Distribution of CO2 in the Atmosphere Student interviews Question 7 (Appendix 1) was administered to nine students during validation interviews. Option 5 (Appendix 1) was added in the third interview. Because none of the interviewed students had taken a climate science class before, they based on high school science, introductory university chemistry, or what they’ve heard from the media.  Only one of the nine students interviewed chose options 1 or 5. Both of these options are meant to represent the misconception that greenhouse gases, or CO2, is trapped below the ozone layer. A B A B 41  Student 88 chose option 5, saying that greenhouse gases are “lighter than the air, and so it will compose the upper level of it.”  This was not the expected justification for these answers, however, Shepardson et al. (2009) found that the prevalence of the misconception that CO2 is trapped below the ozone layer has decreased in the past few decades. Option 3 is also meant to show a distinct layer of “trapped” CO2, but options 1, 3, and 5 did not capture all of the students who held misconceptions that that CO2 is “trapped” below something inside the atmosphere. Student 89 thought greenhouse gases would be “trapped” by radiation – but preferentially at the poles:  “Greenhouse gases like getting trapped there, because more radiation? Like more …um yeah, radiation. Maybe radiation.”  Student 87 considered choosing that 1 and 5 because these options allowed for the “room” for rays to “bounce back” to earth, demonstrating that they also held the misconception that global warming is caused by the sun rays reflecting off the earth, and bouncing back to earth from a layer of greenhouse gases:   “is like the sun radiation comes in and then like, it either like comes out or it bounces off the gas and comes back down like it stays, like nice and insulating, right? … Does this allow for the sun’s radiation to come in and go back down?”  In conclusion, these options aren’t capturing the misconceptions they were intended to. Instead, they may be capturing the misconception that CO2 rises because it is less dense than air, or that CO2 forms a layer because sun rays bounce off of the layer. Four of the interviewed students chose option 2, because the students said greenhouse gases should occur near high population centers, where more human emissions occur. For example, student 81 said: “More where people live, because we’ve been talking about carbon emissions, and there are more people in these areas.”  Three of the interviewed students thought that there should be a higher concentration of CO2 above South America, because there are also human greenhouse gas emissions there. 42  Therefore, this change should be made to ensure this question captures all students who believe CO2 concentration in the atmosphere is correlated to where the emissions are. Many students found that options 3 and 4 were similar. Two chose option 4 simply because it looked the “least severe” or “prettiest” of all the options – so more effort should be made to ensure the other options have symmetry similar to option 4. Several students chose option 4 because CO2 is “more dense” than air – even though the option is meant to show CO2 evenly distributed throughout the atmosphere. Some students chose not to pick option 3 because the concentration looked high, even though all options are supposed to show the same total concentration of greenhouse gases. For example, student 84 described option 3 as: “I don’t think it’s that bad, right now.” At least two students confused CO2 with visible pollution when considering option 2 or 3. For example, when justifying choosing of option 2, student 85 said: “China is well one of the biggest contributor to the carbon dioxide, and it says, the distribution of carbon gases, so I think it should be thicker in front of, like above Asia. But for Canada, like the sky is so blue, and you know the air is relatively fresh” Many interviewed students were confused by option 6, which is meant to represent the misconception that greenhouse gases cause holes in the ozone layer – which occur at the poles. However, no students mentioned ozone in reference to any of these options, and most quickly discounted this option with comments such as, “nobody drives up there, so how would you even get emissions up there?” (Student 83).  This question still needs improvement. Students found the distinctions between options 1 and 5 and between options 3 and 4 were minor. To improve the distinction between the diagrams, each of the distribution images should contain the same number of dots (same average concentration of CO2). However, it is difficult to do this while making a clear distinction between the different distribution types. For example, a layer of CO2 at high altitude will appear less dense if it uses the same number of dots as a low altitude layer, because the circumference of the circle at high altitude is larger than the low altitude circle.  43  Pre-tests and post-tests Fewer than 30% of students select the correct answer (option 4, appendix 1) on either pre-test or post-test. The low frequency may be because a few of the options look similar to each other. Therefore, for the purposes evaluating this question, options 3 and 4 are grouped because as both have CO2 higher near the surface and options 1 and 5 are grouped because both have highest concentrations away from the surface.  In the post-test, students less frequently picked options 1 or 5, 2, and 6 (Figure 24). The decrease in picking options 2 and 6 demonstrates that students have an increased understanding that CO2 is well mixed throughout the atmosphere. The increase in frequency of options 3 and 4 demonstrates that students have an increased understanding both that CO2 is evenly distributed around the globe, and that CO2 is more concentrated at the earth’s surface than at higher altitudes (Figure 24).  Figure 24. Student answers to CO2 distribution question – with similar distribution options grouped together.  44  SUMMARY AND CONCLUSION This study aimed to address five questions about climate science education. First, the CO2 trajectory question and feedback questions from previous EOSC 340 tests were validated and changed. The changes to the CO2 question made it more difficult for students, while the changes to the feedback questions had little effect on student performance. Secondly, four additional questions were developed and validated to test student misconceptions about climate systems. The carbon reservoir and inflow ranking questions measured the knowledge gains about carbon cycle components from the pre-test to the post-test, but the carbon outflow ranking question did not appropriately measure knowledge gains. The CO2 distribution question could be a good knowledge assessment tool, but needs further improvement before it can be used to accurately assess student knowledge. Third, pre-tests and post-tests demonstrated that knowledge was gained about climate system dynamics, feedback loops, carbon reservoirs and flows, and greenhouse gas distribution in the atmosphere. Fourth, Retention test demonstrated that students still remembered some concepts of feedback mechanisms one year after the course, and lost no knowledge specific to mass balance in system dynamics. Lastly, Students remembered both how to construct feedback loops and carbon flow trajectories, and remembered specific mechanisms of climate feedback loops, although some mechanisms, such as Planck’s feedback and lapse rate were forgotten after one year.  Suggestions for Changes to Assessment Questions Presenting the outflow in the CO2 trajectory question as a dot rather than a line makes the question more difficult for students, and doesn’t increase the frequency of students correctly drawing outflow trajectories that don’t unrealistically increase after 2010. If prompted with follow-up questions specifically asking about the processes leading to the trajectories they drew, students are more likely to incorporate knowledge of climate processes into their inflow and outflow trajectories, and are more likely to draw trajectories with correct mass balance. Therefore, it is recommended that short, written follow-up questions accompany the CO2 trajectory question. The extra arrows and guiding words added to the feedback question seemed to increase the frequency of students drawing un-closed feedback loops, and did not otherwise improve student answers in pre-tests or post-tests. 45  Therefore, the extra arrows version should not be used in future assessments. The validated version of the reservoir ranking appropriately question measures students’ knowledge of carbon reservoirs. However, students may be confused about whether the current inflow and outflow questions ask for “total” or “net” flows, so the wording of the questions should be changed to make the intention more clear. The outflow question is not useful as written, as it is not clear whether it measures student knowledge, or familiarity of outflow processes. The pre-tests and post-tests show that the CO2 distribution question measures knowledge gained during the course, but the some of the diagrams should be redrawn as students in interviews had difficulty distinguishing between options 1 and 5, and options 3 and 4. Recommendations for Teaching Students seem able to understand the concepts of mass-balance even prior to taking EOSC 340, and most science students enrolled in EOSC 340 probably have a sufficient background to allow them to interpret and draw graphs which maintain the conservation of mass. However, student struggled to quantify and compare the effects and the capacity of natural and anthropogenic mechanisms of the carbon cycle – which is shown both in the unrealistic inflow and outflow trajectories drawn for the CO2 trajectory question, as well as the tendency of students to overestimate the size of anthropogenic carbon flows in the ranking questions. For future teaching, more emphasis could be focused on learning about the size and capacities of natural and anthropogenic processes in the carbon cycle and how to apply these to projections, in addition to simple models of stocks and flows, and interpreting graphs.  Students gain knowledge about several specific mechanisms that cause feedbacks in the climate system, including silicate weathering and ice albedo. However, it appears that students gain more knowledge about some processes than others. Very few students were confident enough about the role of clouds, ocean outgassing, and lapse rate in climate feedbacks to use these as answers to the post-test and retention test feedback questions, so more emphasis could be placed on these concepts in future teaching.   46  REFERENCES CITED Adams, W.K., and Wieman, C.E. (2011). Development and validation of instruments to measure learning of expert‐like thinking. International Journal of Science Education, 33(9), 1289–1312. doi:10.1080/09500693.2010.512369 Assaraf, O.B.Z., and Orion, N. (2005). Development of system thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518–560. doi:10.1002/tea.20061 Blunt, M.J., and Blizard, P. J. (1975). Recall and retrieval of anatomical knowledge. British Journal of Medical Education, 9, 255–263. Bord, R. J., Fisher, A., and O’Connor, R. E. (1998). Public perceptions of global warming: United States and international perspectives. Climate Research, 11(1), 75–84. Bostrom, A., Morgan, M. G., Fischhoff, B., and Read, D. (1994). What do people know about global climate change? 1. Mental models. Risk Analysis, 14(6), 959–970. doi:10.1111/j.1539-6924.1994.tb00065.x Brewer, C. A., and Beiswenger, J. M. (1993). Carbon dioxide and the greenhouse effect: A problem evaluation activity. The American Biology Teacher, 55(4), 238–240. doi:10.2307/4449641 Cronin, M. A., Gonzalez, C., and Sterman, J. D. (2009). Why don’t well-educated adults understand accumulation? A challenge to researchers, educators, and citizens. Organizational Behavior and Human Decision Processes, 108(1), 116–130. doi:10.1016/j.obhdp.2008.03.003  Custers, E. J. F. M. (2008). Long-term retention of basic science knowledge: a review study. Advances in Health Sciences Education, 15(1), 109–128. doi:10.1007/s10459-008-9101-y Dey, E. L. (1997). Working with low survey response rates: The efficacy of weighting adjustments. Research in Higher Education, 38(2), 215–227. Donovan, J. C., Salzman, L. F., and Allen, P. Z. (1969). Patterns of learning in medical school. Journal of Medical Education, 44(7), 589–594. DuBois, A. B., Nemir, P. Jr., Schumacher, C. F., and Hubbard, J. P. (1969). Graduate medical education in basic sciences. Journal of Medical Education, 44(11), 1035–1043. Dutt, V., and Gonzalez, C. (2012). Decisions from experience reduce misconceptions about climate change. Journal of Environmental Psychology, 32(1), 19–29. doi:10.1016/j.jenvp.2011.10.003 Ebert-May, D., Batzli, J., and Lim, H. (2003). Disciplinary research strategies for assessment of learning. BioScience, 53(12), 1221–1228. doi:10.1641/0006-3568(2003)053[1221:DRSFAO]2.0.CO;2 Gottesman, A. J., and Hoskins, S. G. (2013). CREATE cornerstone: introduction to scientific thinking, a new course for STEM-interested freshmen, demystifies scientific thinking through analysis of scientific literature. Cell Biology Education, 12(1), 59–72. doi:10.1187/cbe.12-11-0201 47  Gowda, M., Fox, J., and Magelky, R. (1997) Students’ understanding of climate change: insights for scientists and educators. Bulletin of the American Meteorological Society, 78(10), 2232–2240. Intergovernmental Panel on Climate Change, Climate Change 2007: The physical science basis: contribution of working group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, S. Solomon et al., Eds. Cambridge Univ. Press, Cambridge. Kastrinos, W. (1965). A study of the retention of biological facts by high school biology students. Science Education, 49, 487–491. Maher, J. M., Markey, J. C., and Ebert-May, D. (2013). The other half of the story: effect size analysis in quantitative research. Cell Biology Education, 12(3), 345–351. doi:10.1187/cbe.13-04-0082 McCaffrey, M. S., and Buhr, S. M. (2008). Clarifying climate confusion: addressing systemic holes, cognitive gaps, and misconceptions through climate literacy. Physical Geography, 29(6), 512–528. doi:10.2747/0272-3646.29.6.512 Niebert, K., and Gropengießer, H. (2011). “CO 2 causes a hole in the atmosphere”: using laypeople’s conceptions as a starting point to communicate climate change. In W. Leal Filho (Ed.), The Economic, Social and Political Elements of Climate Change (pp. 603–622). Berlin, Heidelberg: Springer Berlin Heidelberg. Retrieved from http://link.springer.com/10.1007/978-3-642-14776-0_37 Pala, Ö., and Vennix, J. A. M. (2005). Effect of system dynamics education on systems thinking inventory task performance. System Dynamics Review, 21(2), 147–172. doi:http://dx.doi.org.ezproxy.library.ubc.ca/10.1002/sdr.310 Porter, D., Weaver, A. J., and Raptis, H. (2012). Assessing students’ learning about fundamental concepts of climate change under two different conditions. Environmental Education Research, 18(5), 665–686. doi:10.1080/13504622.2011.640750 Reynolds, T. W., Bostrom, A., Read, D., and Morgan, M. G. (2010). Now what do people know about global climate change? Survey studies of educated laypeople. Risk Analysis, 30(10), 1520–1538. doi:10.1111/j.1539-6924.2010.01448. Rickard, H. C., Rogers, R., Ellis, N. R., and Beidleman, W. B. (1988). Some retention, but not enough. Teaching of Psychology, 15(3), 151–152. Riebeek, H. (2011) The Carbon Cycle. Retrieved from NASA’s Earth Observatory website: http://earthobservatory.nasa.gov/Features/CarbonCycle/ Rodriguez, R., Campos-Sepulveda, E., Vidrio, H., Contreras, E., and Valenzuela, F. (2002). Evaluating knowledge retention of third-year medical students taught with an innovative pharmacology program. Academic Medicine June 2002, 77(6), 574–577. 48  Shepardson, D. P., Niyogi, D., Choi, S., and Charusombat, U. (2010). Students’ conceptions about the greenhouse effect, global warming, and climate change. Climatic Change, 104(3-4), 481–507. doi:10.1007/s10584-009-9786-9 Sterman, J. D. (2008). Risk communication on climate: mental models and mass balance. Science, 322(5901), 532–533. doi:10.1126/science.1162574 Sterman, J. D., and Sweeney, L. B. (2002). Cloudy skies: assessing public understanding of global warming. System Dynamics Review, 18(2), 207. Sterman, J. D., and Sweeney, L. B. (2007). Understanding public complacency about climate change: adults’ mental models of climate change violate conservation of matter. Climatic Change, 80(3-4), 213–238. doi:http://dx.doi.org.ezproxy.library.ubc.ca/10.1007/s10584-006-9107-5                         49  Appendix 1 Validated assessment questions. Question 1A. CO2 trajectory – unvalidated –line (CU) Imagine a scenario in which the stock of CO2 in the atmosphere rises then stabilizes at about 1000 Gton C by the year 2100, as in the figure below.  “Gton C” means “gigatons of carbon”.  Each gigaton is one billion tons.    The graph below shows human carbon emissions up to the year 2010 (inflow) and removal of CO2 from the atmosphere by natural processes (outflow).  Given the scenario above, with CO2 stabilizing by about 2100, on the plot below, please sketch: a. Your estimate of the likely future inflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100 b. Your estimate of the likely future outflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100                  (Continue each of the lines below out to the year 2100)     Inflow Outflow Stock (Gton C in atmosphere) Flows (Gton C per year) 50  Question 1B. CO2 trajectory -validated-line (CVL)  Imagine a scenario in which the stock of CO2 in the atmosphere rises then stabilizes at about 1000 Gton C by the year 2100, as in the figure below.  “Gton C” means “gigatons of carbon”.  Each gigaton is one billion tons.    The graph below shows human carbon emissions up to the year 2010 (inflow) and the net removal of CO2 from the atmosphere by natural processes up to the year 2010 (outflow). Given the scenario above, with CO2 stabilizing by about 2100, on the plot below, please sketch: c. Your estimate of the likely future inflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100 d. Your estimate of the likely future outflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100          Inflow Outflow Stock (Gton C in atmosphere) Flows (Gton C per year) Inflo utflo 51  Question 1C. CO2 trajectory – validated – dot (CVD)  Imagine a scenario in which the stock of CO2 in the atmosphere rises then stabilizes at about 1000 Gton C by the year 2100, as in the figure below.  “Gton C” means “gigatons of carbon”.  Each gigaton is one billion tons.    The graph below shows human carbon emissions up to the year 2010 (inflow) and the net removal of CO2 from the atmosphere by natural processes (outflow) for the year 2010 only. Given the scenario above, with CO2 stabilizing by about 2100, on the plot below, please sketch: e. Your estimate of the likely future inflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100 f. Your estimate of past outflow (before 2010) that would explain the CO2 accumulation from 1900-2010. g. Your estimate of the likely future outflow (through 2100) that would result in stabilization of CO2 in the atmosphere by 2100         Inflow Outflow Stock (Gton C in atmosphere) Flows (Gton C per year) 52  Question 1 i-iv. CO2 Trajectory Follow-up questions i. What causes the slope of the line in the stock graph to change between 1900 and 2010? ii. What caused the slope of the line in the stock graph to change between 2010 and 2100? iii. What specific processes caused the changes to the slope of the inflow line that you drew? iv. What specific processes caused the changes to the slope of the outflow line that you drew?  Question 2A. feedback – not validated – amplifying (FUN1).  Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.  From this starting point, create a sequence of events on Earth that results in global temperatures getting colder and colder and colder.       Question 2B. Feedback – validated – no extra arrows (FVN1) – amplifying feedback  Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.   Given these first two steps, create a sequence of events (natural or human-caused) on Earth that results in global temperatures getting colder and colder and colder. Use as many boxes and arrows as you need to complete the cycle.   Question 2C. Feedback – validated –extra arrows (FVA1) – amplifying feedback Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.   Given these first two steps, create a sequence of connected events (natural or human-caused) on Earth that results in global temperatures getting colder and colder and colder. Use as many boxes and arrows as you need to complete the cycle; the first box is provided for you.      53  Question 3A. Feedback – not validated – stabilizing feedback (FUN2)  Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.  From this starting point, create a sequence of events on Earth that results in global temperatures getting WARMER and returning to a stable temperature again.     Question 3B. Feedback – validated – no extra arrows (FVN2) – stabilizing feedback Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.  Given these first two steps, create a sequence of events (natural or human-caused) on Earth that results in global temperatures returning to a stable temperature again. Use as many boxes and arrows as you need to complete the cycle.    Question 3C. Feedback – validated –extra arrows (FVA2) – stabilizing feedback Imagine a scenario in which, for some reason, global temperature gets colder.  This causes other things to happen.  Given these first two steps, create a sequence of connected events (natural or human-caused) on Earth that results in global temperatures returning to a stable temperature again. Use as many boxes and arrows as you need to complete the cycle; the first box is provided for you.        54   Question 4.Validated reservoir ranking question (same as un-validated version) Carbon is stored in many different places and forms. These places are often called “reservoirs”. Rank the following carbon reservoirs from largest (containing the most total Carbon) to smallest (containing the least total Carbon):  A. carbon in CO2 in the atmosphere B. carbon dissolved in ocean water C. carbon in living biomass (plants + animals) D. carbon in all rocks  E. carbon in fossil fuels  LARGEST (contains the most total carbon)      SMALLEST (contains the least total carbon)  Question 5A. Un-validated inflow ranking question Rank the following processes by how much carbon they ADD to the atmosphere each year, from largest flow (adds the MOST carbon to the atmosphere each year) to smallest flow (adds the LEAST carbon to the atmosphere each year).  A. Fossil fuel emissions from humans B. CO2 coming out of the oceans C. CO2 coming out of volcanoes D. Respiration by plants and decomposition in soil  E. Deforestation and land use change Adds the MOST carbon to the atmosphere per year      Adds the LEAST carbon to the atmosphere per year       55   Question 5B. Validated inflow ranking question. Rank the following processes by how much carbon they ADD to the atmosphere, on average, each year, from largest flow (adds the MOST carbon to the atmosphere each year) to smallest flow (adds the LEAST carbon to the atmosphere each year).  F. Fossil fuel emissions from humans G. CO2 coming out of the oceans H. CO2 coming out of volcanoes I. Respiration by plants and decomposition in soil J. Deforestation by humans Adds the MOST carbon to the atmosphere per year      Adds the LEAST carbon to the atmosphere per year  Question 6A. un-validated outflow ranking question Rank the following processes by how much carbon they TAKE OUT of the atmosphere each year, from largest flow (takes the MOST carbon out of the atmosphere each year) to smallest flow (takes the LEAST carbon out of the atmosphere each year).  A. CO2 taken up in the process of weathering rocks B. CO2 taken up by plants through photosynthesis C. CO2 dissolving in ocean water Takes out the MOST carbon from the atmosphere per year    Takes out the LEAST carbon from the atmosphere per year  Question 6B. Validated outflow ranking question Rank the following processes by how much carbon they TAKE OUT of the atmosphere, on average, each year, from largest flow (takes the MOST carbon out of the atmosphere each year) to smallest flow (takes the LEAST carbon out of the atmosphere each year).  D. CO2 taken up in the process of weathering rocks E. CO2 taken up by plants through photosynthesis F. CO2 dissolving in ocean water Takes out the MOST carbon from the atmosphere per year    56  Takes out the LEAST carbon from the atmosphere per year Question 7A. un-validated CO2 distribution question  Which of the diagrams below best represents the distribution of greenhouse gases in earth’s atmosphere? Circle your answer.  Question 7B. Validated CO2 distribution question Which of the diagrams below best represents the distribution of greenhouse gases in earth’s atmosphere? Circle your answer.     57  Appendix 2.Examples of answers to feedback questions   Code marks All sequential steps are actually related 0 Loop leads back to decreasing T 0 All steps involve internally consistent timescales +1 an incorrect statement 0 Total 1 “extra step” 1 Example 1. A typical answer to the negative feedback question which does not have sequential steps and does not lead to a continued decrease in temperature. Coding scheme included. The “extra step” is “sea level becomes constant” between “colder temp” and “sea level decrease because ice is retained”. Sea level decrease is not assumed to be an incorrect statement because if snow/ice retained includes land-based snow and ice, it could result in sea level decrease.  Code marks All sequential steps are actually related +1 Loop leads back to counteract decreasing T +1 All steps involve internally consistent timescales 0 an incorrect statement -1 Total 1 Example 2. A typical answer to the positive feedback loop using “humans” as a feedback mechanism. The effects of anthropogenic fossil fuel use have effects result in effects in the global temperature that have longer time scales than human’s tendency to turn the heater up or down. In addition, the increase or decrease in the temperature on the heater will not correlate to the opposite change in the global temperature, thus the question is also allocated an “incorrect statement.  58   Example 3. A typical answer to the positive feedback question using “plants dying” as a feedback mechanism.   

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.52966.1-0074570/manifest

Comment

Related Items