UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Strategies for promoting real-world connections in problem solving for introductory physics Martinuk, Mathew Alexander 2012

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2012_fall_martinuk_mathew.pdf [ 5.54MB ]
Metadata
JSON: 24-1.0073082.json
JSON-LD: 24-1.0073082-ld.json
RDF/XML (Pretty): 24-1.0073082-rdf.xml
RDF/JSON: 24-1.0073082-rdf.json
Turtle: 24-1.0073082-turtle.txt
N-Triples: 24-1.0073082-rdf-ntriples.txt
Original Record: 24-1.0073082-source.json
Full Text
24-1.0073082-fulltext.txt
Citation
24-1.0073082.ris

Full Text

  STRATEGIES FOR PROMOTING REAL-WORLD CONNECTIONS IN PROBLEM SOLVING FOR INTRODUCTORY PHYSICS by MATHEW ALEXANDER MARTINUK B.A.Sc., The University of British Columbia, 1999 M.S., Stanford University, 2001  A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  DOCTOR OF PHILOSOPHY  in  THE FACULTY OF GRADUATE STUDIES (Physics)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2012 © Mathew Alexander Martinuk, 2012   ii ABSTRACT  It is well known that students’ beliefs about the nature of knowledge and learning in physics affect their motivation and learning in this subject.  In this thesis I examine a course transformation that presented the entire course in terms of real- world circumstances and problems which was intended to promote students’ perception of the relevance of physics to the real world and to enable them to develop problem-solving skills for addressing complex real-world problems.  By examining course-wide surveys I demonstrate that the intended improvement in students’ perception of the relevance of physics did not occur, suggesting that teaching physics in a real-world context is not, in itself, sufficient for improving students’ belief in the relevance of physics.  Two interview studies reveal some of the features that students report are important in their perception of the relevance of physics to the real world.  I use the theoretical framework of epistemological framing and resources to study students’ response to complex real-world problems that were intended to promote students’ use of real-world knowledge and enable development of expert-like problem-solving skills.  I argue that students’ use of real-world knowledge within a physics context are opportunities for them to develop a more favorable attitude towards physics in general and develop a coding scheme for identifying these instances.  This study demonstrates that students’ use of their real-world knowledge in physics is highly correlated with their framing of their activity as conceptual (rather than procedural) discussion.  In addition, I demonstrate that the course’s structured problem solving method is not effective at promoting conceptual discussion at appropriate times during the problem-solving process.   iii PREFACE  This research was conducted under approval from UBC’s Behavioral Research Ethics Board under study IDs H07-01702, H08-01676, and H09-02793.     iv TABLE OF CONTENTS Abstract ............................................................................................................................ ii	
   Preface ............................................................................................................................. iii	
   Table of Contents ............................................................................................................. iv	
   List of Tables .................................................................................................................... xi	
   List of Figures ................................................................................................................... xv	
   Acknowledgements ........................................................................................................ xvii	
   Dedication .................................................................................................................... xviii	
   1	
   Introduction ................................................................................................................ 1	
   1.1	
   Motivation:  Useful Education .............................................................................. 1	
   1.2	
   Changes to Physics 100 ........................................................................................ 2	
   1.2.1	
   Integrating Physics and Real-World Knowledge ............................................. 2	
   1.2.2	
   Beliefs About Relevance of Physics ................................................................ 3	
   1.2.3	
   Problem Solving ............................................................................................. 4	
   1.3	
   Research Approach and Research Questions ........................................................ 5	
   1.3.1	
   Research on Beliefs About Relevance Of Physics ........................................... 5	
   1.3.2	
   Research on Integrating Physics and Real-World Knowledge ......................... 6	
   1.3.3	
   Research on Conceptual Discussion During Problem-Solving ........................ 6	
   1.4	
   Dissertation Overview .......................................................................................... 7	
   1.4.1	
   Study of Changes in Students' Beliefs ............................................................. 8	
   1.4.2	
   Exploratory Post-Course Interviews ................................................................ 9	
   1.4.3	
   Structured Real-World Connections Interviews ............................................ 10	
   1.4.4	
   Epistemological Framing and Real-World Connections in Structured Group Problem-Solving ...................................................................................................... 10	
   2	
   Development of Physics 100 ..................................................................................... 13	
   2.1	
   Background of Physics 100 ................................................................................. 13	
   2.2	
   Motivation for Changes ...................................................................................... 13	
   2.2.1	
   Students’ Understanding of Societally Relevant Issues .................................. 14	
   2.2.2	
   Retention ..................................................................................................... 14	
   2.2.3	
   Perception of Relevance of Physics .............................................................. 15	
   2.2.4	
   Decline in CLASS Survey Scores .................................................................. 15	
   2.2.5	
   Real-World Problem-Solving Skills ............................................................... 16	
   2.3	
   Goals of Changes ............................................................................................... 16	
    v 2.4	
   Changes to Physics 100 ...................................................................................... 17	
   2.4.1	
   Everyday Context ......................................................................................... 17	
   2.4.2	
   Course Content ............................................................................................ 18	
   2.4.2.1	
   Thermal Physics and Climate Change .................................................... 19	
   2.4.2.2	
   Air Resistance and Energy in Transportation .......................................... 20	
   2.4.2.3	
   Chemical and Metabolic Energy ............................................................ 21	
   2.4.2.4	
   Power in Circuits ................................................................................... 21	
   2.4.2.5	
   Eliminated Topics .................................................................................. 21	
   2.4.3	
   Problem-Solving Skills .................................................................................. 22	
   2.4.3.1	
   Modeling and Assumptions ................................................................... 23	
   2.4.4	
   Structured Problem-Solving .......................................................................... 24	
   2.4.4.1	
   Step 1: Interpret the Problem ................................................................. 25	
   2.4.4.2	
   Step 2: Identify Relevant Physics ............................................................ 26	
   2.4.4.3	
   Step 3 & 4: Create a Physics Model ....................................................... 26	
   2.4.4.4	
   Step 5: Solve the Problem ...................................................................... 28	
   2.4.4.5	
   Step 6:  Error-Checking and Sensemaking .............................................. 28	
   2.4.4.6	
   Complete Physics 100 Problem Solving Strategy .................................... 29	
   2.4.4.7	
   On the Lack of a Planning Step .............................................................. 31	
   2.4.4.8	
   Scoring .................................................................................................. 32	
   2.4.5	
   Changes to Recitations ................................................................................. 34	
   2.4.5.1	
   Context-Rich Recitations ....................................................................... 35	
   2.4.5.2	
   Group Structure and Roles ..................................................................... 36	
   2.4.5.3	
   Workshops for Introducing Problem-Solving Skills ................................. 37	
   2.4.5.4	
   Everyday Context ................................................................................... 39	
   2.4.5.5	
   Motivation for Calculation ..................................................................... 40	
   2.4.5.6	
   Two Versions of Each Tutorial ............................................................... 40	
   2.4.5.7	
   Solutions as Worked Examples .............................................................. 42	
   2.4.6	
   Changes to Lecture ....................................................................................... 43	
   2.4.7	
   Group Research Project ............................................................................... 44	
   3	
   Study of Changes in Students’ Beliefs About Physics .................................................. 45	
   3.1	
   Goal and Research Question .............................................................................. 45	
   3.2	
   Methodology ...................................................................................................... 45	
   3.3	
   CLASS Survey Results ......................................................................................... 47	
    vi 3.4	
   Analysis of Pre-Course Scores ............................................................................. 49	
   3.5	
   Analysis of Lack of Change in 2007 .................................................................... 51	
   3.5.1	
   Differences in Population due to Recruitment Method ................................. 52	
   3.5.2	
   Faculty Changes ........................................................................................... 52	
   3.6	
   Analysis of Decline in 2009 ................................................................................ 54	
   3.6.1	
   Instructor Changes ....................................................................................... 54	
   3.6.2	
   Student Demographics ................................................................................. 56	
   3.6.3	
   Timing differences ........................................................................................ 56	
   3.7	
   Summary ............................................................................................................ 58	
   4	
   Exploratory Post-Course Interviews ............................................................................ 59	
   4.1	
   Goal and Research Questions ............................................................................. 59	
   4.2	
   Selection of Participants ..................................................................................... 60	
   4.3	
   Interview Protocol .............................................................................................. 60	
   4.4	
   A Methodological Comment ............................................................................... 63	
   4.5	
   Interview Results ................................................................................................ 64	
   4.5.1	
   Diverse Interpretation of “Reasoning Skills Used to Understand Physics” ..... 64	
   4.5.1.1	
   Reasoning Skills as Problem Solving ...................................................... 64	
   4.5.1.2	
   Reasoning Skills as Physics Content ....................................................... 66	
   4.5.1.3	
   Reasoning Skills as Common Sense ....................................................... 67	
   4.5.1.4	
   Reasoning Skills as Exam-Taking Strategies ............................................ 68	
   4.5.2	
   Conceptions of Physics in the Real World .................................................... 69	
   4.5.2.1	
   Physics as Relevant Environmental Issues .............................................. 69	
   4.5.2.2	
   Physics as Relevant vs. Physics as Worth Noticing ................................. 71	
   4.5.2.3	
   Physics as Calculations .......................................................................... 71	
   4.5.3	
   Factors Influencing Perception of Relevance ................................................ 72	
   4.5.3.1	
   Personal Perspective on Real-World Relevance ..................................... 72	
   4.5.3.2	
   Interpretation of Realism of Tutorials ..................................................... 73	
   4.5.4	
   Summary ...................................................................................................... 74	
   5	
   Structured Real-World Relevance Interviews ............................................................. 75	
   5.1	
   Goal and Research Questions ............................................................................. 75	
   5.2	
   Methodology ...................................................................................................... 76	
   5.2.1	
   Interview Cohort .......................................................................................... 76	
   5.2.2	
   Interview Protocol ........................................................................................ 77	
    vii 5.3	
   Qualitative Analysis and Results ......................................................................... 79	
   5.3.1	
   Judgment Based on Explicit Context ............................................................. 80	
   5.3.2	
   Diverse Definitions of Real-World Connections ........................................... 80	
   5.3.3	
   Real-World Triggers ..................................................................................... 81	
   5.3.3.1	
   Examples of Trigger Coding ................................................................... 82	
   5.3.3.2	
   Categories of Real-World Triggers ......................................................... 84	
   5.3.4	
   Discussion ................................................................................................... 90	
   5.4	
   Quantitative Analysis and Results ....................................................................... 90	
   5.4.1	
   Coding of Problems ...................................................................................... 91	
   5.4.2	
   Coding of Responses to Interview Problems ................................................. 92	
   5.4.3	
   Statistical Analysis of Codes ......................................................................... 93	
   5.4.4	
   Discussion ................................................................................................... 96	
   5.5	
   Summary ............................................................................................................ 97	
   6	
   Epistemological Framing and Real-World Connections in Structured Group Problem- Solving ............................................................................................................................ 98	
   6.1	
   Introduction ........................................................................................................ 98	
   6.1.1	
   Research Questions ...................................................................................... 99	
   6.1.2	
   Research Context ....................................................................................... 100	
   6.2	
   Background and Theoretical Framework .......................................................... 101	
   6.2.1	
   Resources and Framing .............................................................................. 101	
   6.2.1.1	
   Review of Scherr and Hammer Coding Scheme ................................... 103	
   6.2.2	
   Real-World Connections ............................................................................ 106	
   6.2.2.1	
   Definition of Real World Connection .................................................. 106	
   6.2.2.2	
   The Importance of Studying RWC in Physics ....................................... 107	
   6.2.2.3	
   Previous Work on Examining Students' RWC ...................................... 110	
   6.2.2.4	
   My Approach ...................................................................................... 112	
   6.2.3	
   Conceptual Discussion During Problem-Solving ........................................ 112	
   6.3	
   Methodology .................................................................................................... 115	
   6.3.1	
   Overall Analysis Strategy ............................................................................ 115	
   6.3.1.1	
   On the Role of Epistemological Framing .............................................. 117	
   6.3.1.2	
   Cohort ................................................................................................. 117	
   6.3.2	
   Coding Problem-Solving Prompts ............................................................... 119	
   6.3.3	
   Validation of Audio-Only Coding of Epistemological Framing .................... 120	
    viii 6.3.3.1	
   Data .................................................................................................... 121	
   6.3.3.2	
   Methodology ....................................................................................... 121	
   6.3.3.3	
   Results ................................................................................................. 123	
   6.3.3.4	
   Limitations of Audio-Only Coding for Epistemological Framing ........... 125	
   6.3.3.5	
   Comments on Scherr and Hammer's Original Coding ......................... 126	
   6.3.3.6	
   Summary ............................................................................................. 127	
   6.3.4	
   Development of Epistemological Framing Coding Scheme ......................... 128	
   6.3.4.1	
   Adaptation to Cohort ........................................................................... 128	
   6.3.4.2	
   Conceptual and Procedural Discussion ............................................... 129	
   6.3.4.3	
   Differences in Worksheet Frame .......................................................... 137	
   6.3.4.4	
   Comparison of Procedural Discussion and Worksheet Frame .............. 139	
   6.3.4.5	
   Other New Frames .............................................................................. 140	
   6.3.4.6	
   Summary of Framing Coding Scheme .................................................. 141	
   6.3.4.7	
   Example of Framing Coding ................................................................. 141	
   6.3.4.8	
   Framing Inter-Rater Reliability Testing ................................................. 146	
   6.3.4.9	
   Limitations of Coding Methodology ..................................................... 149	
   6.3.5	
   Coding Real-World Connections ................................................................ 152	
   6.3.5.1	
   Development of RWC Coding Scheme ................................................ 153	
   6.3.5.2	
   RWC Inter-Rater Reliability Testing ...................................................... 155	
   6.3.5.3	
   Summary of RWC Coding Scheme ...................................................... 156	
   6.3.6	
   Equivalence of Frames and Codes .............................................................. 164	
   6.4	
   Coding Results and Analysis ............................................................................. 164	
   6.4.1	
   Selection of Data ........................................................................................ 164	
   6.4.2	
   Correlation Between Frame and RWC ........................................................ 165	
   6.4.2.1	
   Frequency of RWC by Frame ............................................................... 165	
   6.4.2.2	
   Investigation of Positive RWC in the M Frame ..................................... 167	
   6.4.2.3	
   Examination of RWC by Episode ......................................................... 170	
   6.4.2.4	
   Do the High RWC Groups Engage in More Conceptual Discussion? .... 172	
   6.4.2.5	
   Real-World Connections in the Conceptual Discussion Frame ............ 176	
   6.4.3	
   Correlation Between Prompts and Conceptual Discussion ......................... 182	
   6.4.4	
   Correlation Between Prompts and RWC ..................................................... 184	
   6.4.4.1	
   RWC in the Assumptions Prompt ......................................................... 186	
   6.4.4.2	
   RWC in the Solve Prompt .................................................................... 189	
    ix 6.4.4.3	
   RWC in the Sensemaking Prompt ........................................................ 192	
   6.4.4.4	
   Summary ............................................................................................. 194	
   6.4.5	
   Analysis of Negative RWC ......................................................................... 194	
   6.4.5.1	
   Detail of Tutorial (4 clusters) ................................................................ 197	
   6.4.5.2	
   Motivation for Calculation (6 clusters) ................................................. 198	
   6.4.5.3	
   Physics Vs. Common Sense (1 cluster) ................................................. 201	
   6.4.5.4	
   Relevant Experience (1 cluster) ............................................................ 202	
   6.4.5.5	
   Interpretation of Goal (2 clusters) ......................................................... 204	
   6.4.5.6	
   Vaguely Negative (3 clusters) ............................................................... 204	
   7	
   Discussion and Conclusions .................................................................................... 206	
   7.1	
   Research Question 1: Impact of Physics 100 Reform on Student Beliefs ........... 206	
   7.1.1	
   Discussion ................................................................................................. 206	
   7.1.2	
   Positive Impacts of Course Transformation ................................................. 207	
   7.2	
   Research Question 2: Types of Real-World Connection .................................... 208	
   7.2.1	
   Judgment of Relevance ............................................................................... 208	
   7.2.2	
   Use of Real-World Resources in Physics .................................................... 209	
   7.2.3	
   Discussion ................................................................................................. 209	
   7.3	
   Research Question 3: Factors Influencing Real-World Connections .................. 210	
   7.3.1	
   Factors Affecting Perception of Relevance .................................................. 211	
   7.3.1.1	
   Context ................................................................................................ 212	
   7.3.1.2	
   Consequences ..................................................................................... 213	
   7.3.1.3	
   Quantitative Reasoning ....................................................................... 215	
   7.3.2	
   Factors Affecting Use of Real-World Knowledge ........................................ 217	
   7.3.2.1	
   Epistemic Framing ............................................................................... 218	
   7.3.2.2	
   Problem-Solving Strategy ..................................................................... 219	
   7.3.2.3	
   Generalizing Beyond RWC .................................................................. 220	
   7.4	
   Research Question 4: Influence of Structured Problem-Solving Methods on Conceptual Discussion .............................................................................................. 221	
   7.5	
   Methodological Contributions .......................................................................... 222	
   7.5.1	
   Interpretation of CLASS results ................................................................... 223	
   7.5.2	
   Development of Coding Scheme for Real-World Connections ................... 224	
   7.5.3	
   Extension of Epistemological Framing Coding Scheme ............................... 224	
   7.6	
   Recommendations for Instruction ..................................................................... 225	
    x References .................................................................................................................... 234	
   Appendix A Physics 100 Tutorial Problems ................................................................... 239	
   Appendix B Exploratory Post-Course Interview Protocol ............................................... 344	
   Appendix C Structured Real-World Connections Interview Protocol ............................. 347	
   Appendix D Epistemological Framing Coding Rubric .................................................... 360	
   Appendix E Real-World Connections Coding Rubric ..................................................... 366	
     xi LIST OF TABLES Table 1:  Overall dissertation research questions and sub-questions by chapter. ............... 8	
   Table 2: Summary of the structured problem-solving strategy used in Physics 100.  These are the titles of the problem-solving steps that were used in lecture and recitation. .. 29	
   Table 3: Detail of the structured problem-solving strategy used in Physics 100.  These are the prompts that were written on the recitation worksheets completed by the students.  ................................................................................................................................ 30	
   Table 4:  Marking rubric used for Physics 100 recitation problems. ................................. 34	
   Table 5:  Skill and content focus for the first five Physics 100 recitations.  These recitations were structured as workshops on developing students’ skills for individual steps of the overall problem-solving method. ............................................................................. 39	
   Table 6:  Criteria for personal context and real context tutorials.  Each recitation problem in 2009 was developed in two versions with identical mathematical structure but slightly different cover stories. .................................................................................. 42	
   Table 7:  CLASS response items in the Real-World Connections category.  These items are scored on a 5-point Likert scale:  strongly agree, agree, neutral, disagree, and strongly disagree.  Responses to items 28, 30, and 37 are considered favorable if the student chooses ‘strongly agree’ or ‘agree’.  Responses to item 35 are considered favorable if the student chooses ‘strongly disagree’ or ‘disagree’. ............................................... 46	
   Table 8:  Class sizes and participation rates for the CLASS survey in Physics 100 from 2006 to 2009.  In 2006 participation in the survey was strictly voluntary.  In 2007 through 2009 1% of course grade was given for participation in both the pre- and post-tests. ................................................................................................................. 47	
   Table 9:  Instructor changes and number of CLASS survey participants in Physics 100 from 2006 to 2010. .......................................................................................................... 52	
   Table 10:  Summary of questions in exploratory post-course interviews.  Students were asked to read these questions aloud, comment on whether they agreed or disagreed with them on a 5-point Likert scale, and asked to explain their reasons for their responses.  Responses to questions with a * next to the item number are considered favorable if the student disagrees with the statements. ............................................. 62	
   Table 11:  Summary of structured real-world connection participant demographics.  The last column of the table gives the students’ score on the CLASS survey Real-World Connections category.  A higher number means the student reports more expert-like beliefs about the relationship between physics and the real world. .......................... 77	
   Table 12:  Summary of triggers for real-world connections.  The main trigger categories are listed to the left and the subcategories are listed underneath these.  The number in parentheses after each trigger is the number of times it was coded over the whole data corpus.   Triggers that were generally cited as enabling real-world connections are written in bold font, those that were cited as inhibiting real-world connections are written in italics, and those that could be cited as either enabling or inhibiting real- world connections are written in standard font. ....................................................... 82	
    xii Table 13:  Properties of scientific problems used to probe students’ perception of relevance to the real world.  Columns 2-5 were coded by the researcher.  The Environmental rating is based on the number of environmental triggers mentioned by the participants in study. .......................................................................................... 92	
   Table 14:  Coding scheme for responses to Reality Link Questions.  In each case, a code of 1 indicates the student reported that they perceived little or no real-world connection in that category, a response of 3 indicates a strong real-world connection, and a response of 2 is neutral or mixed. ............................................................................ 93	
   Table 15:  P-values of tests to examine the relationship between problem characteristics on student responses to Reality Link Questions.  *(p < 0.1), **(p< 0.05), ***(p < 0.01).  ................................................................................................................................ 95	
   Table 16:  P-values of Kruskal-Wallis test to examine the impact of the interaction between Everyday Context and Money on real-world-connection questions.   *(p < 0.1),  **(p< 0.05),  ***(p < 0.01). ............................................................................. 96	
   Table 17:  P-values of pairwise Mann-Whitney comparisons of average results between the three groups of problems.  Group 1 is Strong Everyday Context and Money; Group 2 is Strong Everyday Context and No Money; Group 3 is Weak Everyday Context and No Money.    *(p < 0.1), **(p< 0.05), ***(p < 0.01). ................................................. 96	
   Table 18:  Four behavioral clusters and associated epistemological frames.  Scherr and Hammer argue that these behaviors in group problem-solving constitute evidence for students’ current approach towards knowledge and learning. ................................ 104	
   Table 19:  Summary of audio and video episodes coded for validation of audio-only coding of epistemological framing. ........................................................................ 121	
   Table 20:  Inter-rater reliability for study of audio-only vs. video coding of epistemological framing.  Frame transitions were coded to within 5 second accuracy.  The inter-rater reliability was calculated as IRR = 1- (errors) / (total duration coded). .................... 124	
   Table 21:  Example of Conceptual Discussion.  The students’ focus is on figuring out what is going on in the situation and the implications for the force in the rope. .............. 130	
   Table 22: Example of Procedural Discussion.  The students’ focus is on figuring out what they should be calculating. .................................................................................... 134	
   Table 23:  Characteristics of Procedural Discussion and Conceptual Discussion frames.  .............................................................................................................................. 137	
   Table 24:  Summary of epistemological framing coding scheme. .................................. 141	
   Table 25:  Transcript to illustrate Procedural Discussion, Conceptual Discussion, and Off- Topic epistemological frames. ............................................................................... 143	
   Table 26:  Inter-rater reliability testing for epistemological framing coding scheme.  The IRR is calculated as 1 - (total of errors that are greater than 6 seconds) / (total time coded) ................................................................................................................... 149	
   Table 27:  Inter-rater reliability testing for Real-World Connections coding scheme.  The IRR was calculated as 1 - (number of RWC coded more than 6 seconds from the other coders’ RWC) / (total number of RWC coded). ....................................................... 156	
    xiii Table 28:  Types of real-world connection identified during collaborative group problem- solving in introductory physics. ............................................................................. 158	
   Table 29:  Example of Real-World Connection as Interpretation. .................................. 160	
   Table 30:  Example of Real-World Connection as Evaluation. ....................................... 161	
   Table 31: Example of Real-World Connection as Personalization. ................................ 162	
   Table 32:  Example of Real-World Connection as Meta-Statement. ............................... 163	
   Table 33:  Summary of time spent, Real-World Connections, and frequency of real-world connections by epistemological frame. .................................................................. 165	
   Table 34:  Transcript of RWC cluster in the Meta frame.  These students have just read a recitation problem that states the stopping distance for a car traveling 50 km/h is 20 meters and are commenting on the realism of the problem itself during this segment. The “−1” in the RWC column indicates that statement was coded as a negative RWC due to being explicitly negative about the realism of the problem.  A “1” indicates a RWC that is not explicitly negative about the realism of the problem ..................... 168	
   Table 35:  RWC by frame for each episode.  The High group is episodes that have 32 or more RWC.  The Low group is groups that have 18 or fewer RWC. ....................... 171	
   Table 36:  Percentage of RWC in a particular frame that are a certain # of seconds after the previous RWC.  The bottom row indicates the percentage that are more than 30 seconds after the previous RWC. ........................................................................... 177	
   Table 37:  Illustration of RWC clusters in the Conceptual Discussion frame. ................. 179	
   Table 38:  Illustration of RWC in the Worksheet and Procedural Discussion frame. ...... 180	
   Table 39:  Illustration of RWC in the Conceptual Discussion and Off-topic frame.  The students have just calculated an answer and are commenting on it. ....................... 181	
   Table 40:  Aggregate percentage of each problem-solving step spent in each Frame. This is calculated as (Total time in frame during this prompt for all groups) / (Total time spent in this prompt for all groups). ................................................................................. 183	
   Table 41:  The Total number of RWC by prompt for the High and Low RWC categories. The High RWC groups have 32 or more RWC (N=6).  The Low RWC groups have 18 or fewer RWC (N=8). ............................................................................................. 185	
   Table 42:  Example of spontaneous use of real-world knowledge to interpret the result of a calculation. ............................................................................................................ 191	
   Table 43:  Snippet from the Error-Checking and Sensemaking step for one of the Low RWC groups, illustrating the cursory consideration of the sensibility of answers that is typical of the data in this study. ............................................................................. 193	
   Table 44:  Snippet from the Error-Checking and Sensemaking step for one of the High RWC group, illustrating a cursory consideration of the sensibility of answers. ........ 193	
   Table 45:  Number of negative RWC broken down by episode number and problem solving prompt. ...................................................................................................... 196	
   Table 46:  Transcript of negative RWC cluster based on objection to a detail of the tutorial.  xiv These students have just read a recitation problem that states the minimum stopping distance for a car traveling 50 km/h is 20 meters.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC, while a “1” indicates a positive RWC. ........................................................................................................ 197	
   Table 47:   Transcript negative RWC cluster based on objection to a detail of the tutorial. These students are discussing the assumptions for their model.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ......................... 198	
   Table 48:  Transcript of example negative RWC cluster based on objection to the motivation for a calculation.  These students have just read a recitation problem that states that a fridge will be secured in the back of a pickup truck with exactly two ropes.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ....................................................................................................... 199	
   Table 49:  Transcript of example negative RWC cluster based on objection to the motivation for a calculation.  These students have just read a recitation problem in which a doctor has advised the patient to avoid more than 450 Newtons of force on their broken rib.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ................................................................................................ 200	
   Table 50:  Transcript of example negative RWC cluster based on objection to the motivation for a calculation. These students are discussing the assumptions necessary to solve a problem involving moving a fridge.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC, while a “1” indicates a positive RWC.  .............................................................................................................................. 201	
   Table 51:  Transcript of example negative RWC cluster based on objection to using physics instead of common sense.  These students are discussing the assumptions necessary to solve a problem involving moving a fridge.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ...................................... 202	
   Table 52:  Transcript of example negative RWC cluster based on objection to the suggestion that students would have experience with which to judge their answer to a problem that involves moving a fridge, where the truck carrying the fridge may crash. A “−1” in the RWC column indicates a statement that was coded as a negative RWC.  .............................................................................................................................. 203	
   Table 53:  Transcript of example negative RWC cluster based on judgment that the goal of the tutorial is unrealistic.  These students are discussing the recitation’s instruction to consider whether it is “safe to drive home”.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ....................................................... 204	
   Table 54:  Transcript of example vaguely negative RWC cluster.  These students have been flipping through their notes searching for other assumptions to make for their recitation problem.    A “−1” in the RWC column indicates a statement that was coded as a negative RWC. ..................................................................................... 205	
     xv LIST OF FIGURES Figure 1:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey.  The shifts are calculated as %shift = %favorable_post - %favorable_pre for matched pairs of pre/post surveys. ................ 48	
   Figure 2:  Pre-post shift in unfavorable responses for the Overall and Real-World Connections categories on the CLASS survey.  The shifts are calculated as %shift = %unfavorable_post - %unfavorable_pre for matched pairs of pre/post surveys. ........ 48	
   Figure 3: Pre-course Favorable CLASS survey scores for 2006 - 2010 in the Overall and Real-World Connections categories ......................................................................... 50	
   Figure 4:  Pre-course Unfavorable CLASS survey scores for 2006 - 2010 in the Overall and Real-World Connections categories ......................................................................... 50	
   Figure 5:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey for the two sections that had the same instructor for 2006 and 2007. .................................................................................. 53	
   Figure 6:  Pre-post shift in unfavorable responses for the Overall and Real-World Connections categories on the CLASS survey for the two sections that had the same instructor for 2006 and 2007. .................................................................................. 53	
   Figure 7:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey for the sections that had the same instructor each year. ................................................................................................ 55	
   Figure 8:  Pre-post shift in unfavorable responses for the CLASS survey Overall and Real- World Connections categories for the sections that had the same instructor each year.  ................................................................................................................................ 55	
   Figure 9:  Favorable CLASS shift results for the 2009 year, broken down by the week in which students completed their post-test.  In this year, the post-testing period was open for two weeks, instead of the usual one week. ................................................ 57	
   Figure 10: Unfavorable CLASS shift results for the 2009 year, broken down by the week in which students completed their post-test.  In this year, the post-testing period was open for two weeks, instead of the usual one week. ................................................ 57	
   Figure 11:  Main coding schemes and correlation studies in Chapter 6. ........................ 116	
   Figure 12: Summary of reasons for inter–coder error when coding epistemological framing with audio only vs. audio plus video. .................................................................... 125	
   Figure 13:  Boxplot of frequency of RWC in different Frames across the data set (N=14 episodes).  The dark bar shows the median, the box shows the 25th - 75th percentiles, and the whiskers show the range of the data.  Outliers, more than 1.5 times the inter- quartile distance, are marked with circles.  The Triangle symbols mark the average value for each episode. .......................................................................................... 166	
   Figure 14:  Histogram of number of RWC by episode.  The episodes are split into two categories:  The High RWC groups have 32 or more RWC (N=6).  The Low RWC groups have 18 or fewer RWC (N=8). .................................................................... 172	
    xvi Figure 15:  Time spent in each frame by High / Low RWC category.  The High RWC groups have 32 or more RWC (N=6).  The Low RWC groups have 18 or fewer RWC (N=8). .................................................................................................................... 173	
   Figure 16:  Frequency of RWC in each frame by High / Low RWC category.  The High RWC groups have 32 or more RWC (N=6).  The Low RWC groups have 18 or fewer RWC (N=8). ........................................................................................................... 175	
   Figure 17:  Boxplot of number of RWC per group by problem-solving step for the High RWC groups (32 or more RWC, N=6 groups) and the Low RWC groups (18 or less RWC, N=8 groups). ............................................................................................... 185	
   Figure 18:  Boxplot of frequency of RWC by problem-solving step for the High RWC groups (32 or more RWC, N=6 groups) and the Low RWC groups (18 or less RWC, N=8 groups). ......................................................................................................... 187	
   Figure 19:  Total number of RWC as a function of time after the Assumptions prompt by High and Low RWC groups.  The shaded background shows the fraction of groups that are still working in the Assumptions step at that time. ..................................... 188	
   Figure 20:  Total number of RWC as a function of time after the Solve prompt by High and Low RWC groups.  The shaded background shows the fraction of groups that are still working in the Solve step at that time. ................................................................... 189	
      xvii ACKNOWLEDGEMENTS I gratefully acknowledge the support of the supervisors, mentors, collaborators, peers, and students that have made my graduate school experience so pleasurable and successful.  Particular thanks are due to my supervisor, Lorne Whitehead, for giving me the opportunity to work in this field which I have so deeply enjoyed.  I would also like to thank Andrzej Kotlicki, Jim Carolan, Anthony Clarke, Gaalen Erickson, and Chris Waltham for their guidance and mentorship throughout my graduate career. I also owe a deep debt of gratitude to my colleagues in Physics Education Research that helped me to clarify my research and hone my writing and presentation skills.  Particular thanks are due to Joe Redish, Joss Ives, Hunter Close, Eleanor Close, Sam MacKagan, Rachel Scherr, Leslie Atkins, and Ellie Sayre for helping me to become a better researcher and making me feel welcome in the field. If I have forgotten anybody in this brief note, know that the fault is mine and not theirs.   xviii DEDICATION This dissertation is dedicated to my wife Donyne O’Coffey and my son Felix.  Without their love and support this work would not only have been impossible, it would have been meaningless.  They have been my greatest friends, my greatest inspiration, and my greatest teachers, and this work is dedicated to them.   1 1 INTRODUCTION 1.1 Motivation:  Useful Education It is a great tragedy of our modern education system that much of what is learned will never be used outside the classroom it is learned in.  Teachers may teach and students may learn the topics on the course curriculum without ever considering whether the learned knowledge has any application or relevance to the world beyond the classroom walls. Considering the ways that an education might be useful outside the classroom can offer instructors a significant new perspective on their teaching.  In 2007 the instructors of the University of British Columbia’s Physics 100 course (the introductory physics course for non-majors) started asking themselves: What use could an introductory physics course be to a student who doesn’t continue in physics?  This question was very pertinent to their course in particular.  Of the 800 students taught in this course every year, less than half a percent of them would carry on in physics beyond the first-year sequence.  Despite this, the course used the same textbook and the same pedagogical approaches used by the courses for physics majors.  The investigation undertaken by the Physics 100 instructors of how to offer a physics education that would be useful to students outside the classroom led to a comprehensive transformation of the UBC Physics 100 course, and set the stage for this dissertation. In this dissertation, I explore three educational goals that, if achieved, can help formal physics education to reach beyond the classroom: 1. integration of formal physics knowledge with students’ everyday knowledge of physics 2. development of students’ belief in the relevance of physics to the real world, and 3. development of flexible problem-solving skills. In the remainder of this introduction, I will briefly describe the motivation and some of the previous research relevant to the above goals.  Then I will describe my approach to   2 investigating each of these goals, and present the research questions addressed in this thesis.  Finally, I will introduce the studies that I have conducted to address those questions and provide an overview of the entire dissertation. 1.2 Changes to Physics 100 In order to promote students’ perception of the relevance of physics to the real world and to enable them to develop flexible real-world problem-solving skills the Physics 100 course at UBC was substantially transformed in the fall of 2007.  These changes are described in detail in chapter 2, but are summarized briefly here. Several strategies were employed in order to address one or more of the above goals. First, every physics problem in the course was presented in terms of a realistic everyday context.  This included all lecture examples, homework problems, and weekly recitation problems.  Second, context-rich problems were adopted for the weekly recitation sessions [1].  These problems require students to work in groups to solve a single complex problem using a structured problem-solving method, and were redesigned to encourage students to make use of their real-world knowledge in order to attain a solution.  Third, to address the challenge of applying simple physics to real-world problems which are often ill-defined and complex, the course emphasized the role of modeling in problem solving.  The practice of making reasonable assumptions in order to represent a real system in a simplified quantitative representation that is amenable to calculation was demonstrated in lecture and was explicitly emphasized during the recitation sessions.  Finally, the course content was substantially changed in order to enable discussion of the physics of everyday circumstances such as home heating, as well as the physics relevant to social and environmental issues such as energy conservation and climate change.  These changes are described in detail in section 2.4. 1.2.1 Integrating Physics and Real-World Knowledge One way that physics education can be relevant outside the classroom is by enabling students to learn formal physics knowledge that is well-integrated with their everyday knowledge.  Here, “well-integrated” means that the physics concepts are understood in terms of everyday knowledge and vice versa.  Enabling students to   3 connect to their existing ideas about physics helps them to learn:  Indeed, integration into their existing framework of knowledge and experience is what gives new ideas meaning [2]. A wide variety of research has been done on the role of students’ informal and intuitive knowledge in learning physics content.  (for a review, see Ref. [3])  Much of this research focuses on the ways in which students’ informal and intuitive knowledge may prevent the learning of canonical physics concepts.  A common way of dealing with students’ unproductive intuitions is to “confront” the informal knowledge to demonstrate its incorrectness [4]. However, some researchers advocate refinement of students’ intuition rather than refutation of it [5].  Sherin has argued that, when physics knowledge and informal intuition are accessed simultaneously, this creates opportunities for intuition to change and evolve, and he suggests that this process may eventually develop the qualitative intuition of an expert [6].  Pugh, in his research on student use of science information outside the classroom, suggests that “applying learning to real-world situations under the direction of the teacher in class may be the first step to applying learning in one’s everyday experience“ [7]. In order to help students integrate their real-world knowledge and intuitions with their formal physics knowledge, Physics 100 was revised to present every lecture example, homework problem, and recitation problem in an everyday context.  In this dissertation I examine the success of these changes by looking at students’ discussions during the weekly recitation sessions to identify the ways in which they make use of their real-world knowledge.  I also examine some of the factors that influence these real-world connections. 1.2.2 Beliefs About Relevance of Physics Because physics involves the study of matter, energy, and motion, it would seem trivially easy to teach physics that is relevant to the world outside the classroom; it could be argued that physics is relevant to every process in the universe.  However, as Pugh observed in his studies of use of science outside the classroom, “The ability to   4 apply knowledge does not guarantee that students actually will apply their knowledge” [8].  He suggests that the belief in the relevance of science to the outside world is an important precursor to the actual use of science in the outside world [7,8]. More generally, students’ beliefs about the nature of physics and learning have also been shown to impact their choice of learning strategies, their satisfaction in the field, and their subsequent persistence in physics [9-12]. Unfortunately, studies of students’ beliefs about physics have revealed that many novice students do not perceive physics to be meaningfully related to their everyday world [13,14].  Furthermore, those same studies show that students’ beliefs typically become less favorable during their first introductory physics course, which can be attributed to implicit messages students receive about physics from typical treatment in curriculum [15,16].  According to these surveys, the only courses that have been shown to help students to view physics as being more coherent, more connected to the real world, and less about memorization of formulae are courses that have an explicit emphasis on highlighting and discussing students’ and scientists epistemological beliefs [15,17-19]. This underscores the influence that pedagogy can have on students’ beliefs.  The potential to positively affect students’ beliefs motivated the transformation in Physics 100 aimed at improving students’ perception of the relevance of physics.  In this dissertation, I examine students’ perception of the relevance of various physics problems to the real world, both in interview studies and via analysis of their discourse during problem solving. 1.2.3 Problem Solving A third way that physics education can be relevant outside the classroom is by teaching students skills and habits of mind that enable them to solve complex problems.  Teaching such “problem-solving skills” is a commonly cited goal of introductory physics education, and there is a host of research on methods for teaching these skills to students.  Many researchers advocate use of a structured problem-solving method as a means to enable students to develop advanced problem-   5 solving skills and to promote integration of conceptual and procedural knowledge (e.g. [1,20,21]). The Physics 100 course adopted a structured problem-solving strategy for use in lectures and the weekly recitation sessions.  In this dissertation I examine how this mandatory problem-solving strategy influenced students’ use of conceptual and real- world knowledge in discussions during their weekly recitation sessions. 1.3 Research Approach and Research Questions In this section I describe my approach to investigating each of these strategies, and present the research questions addressed in this thesis. It is important to note that, although these studies are conducted within the context of the transformation of the Physics 100 course, and chapter 3 is concerned with evaluating the impact of those changes, that is not the principal focus of this dissertation.  Rather, this work is an exploration, in the spirit of Interpretive Research [22], of the different ways in which students perceive connections between physics and the real world and of the factors that influence such connections. 1.3.1 Research on Beliefs About Relevance Of Physics In this dissertation I examine how students’ beliefs about physics changed over the course of Physics 100.  Specifically, I investigate whether the course transformation had the intended effect of promoting students’ perception of the relevance of physics to the real world. In order to evaluate the effect of the course transformation I rely on the only two sources of information available on the pre-transformation course:  the CLASS survey of student beliefs about physics [14], and the instructors’ recollections of the pre- transformation course.    The CLASS survey assesses students’ beliefs about physics in a wide variety of areas, including their belief in the relevance of physics to the real world, and their beliefs about the nature of knowledge in physics and how it is developed.  This survey was conducted in 2006, the year prior to the transformation,   6 and the measured decline in students’ perception of the relevance of physics to the real world was one of the motivating factors for the course changes. Using these two sources of information, I address the research question:  Research Question 1:  Did the changes to Physics 100 improve the impact of the course on students' beliefs regarding the relevance of physics to the real world? 1.3.2 Research on Integrating Physics and Real-World Knowledge Because of the possibility that working with physics and informal knowledge together may enable better integration between them, I have studied the ways in which students make connections between these two types of knowledge. In prior work, I (and others) have examined students’ connections between formal physics and the real world by looking at their ability to apply physics to written problems that are set in an everyday context [23,24].  Another approach has been to examine teachers’ and students’ mentions of real-world science within a classroom [25]. In my work, I follow Sherin’s approach of examining students’ discourse during problem-solving to identify instances where they make use of their informal intuition and everyday knowledge in concert with physics knowledge to clearly identify the co- activation of real-world and physics knowledge that leads to integration of these two types of knowledge [6].  My dissertation addresses the following two research questions: Research Question 2:  In what ways do students make connections between physics and the real world? Research Question 3:  What influences whether and how students make connections between physics and the real world? 1.3.3 Research on Conceptual Discussion During Problem-Solving The problem-solving strategy used in the Physics 100 recitations includes several steps that are explicitly intended to induce students to make use of their conceptual   7 and qualitative knowledge both prior to and following their main calculations.  As I will describe in Section 6.2.3, several published problem-solving strategies also include steps with the same intention [1,20,21]. These strategies are motivated by studies in differences of how experts and novices solve problems which show that experts are more likely to make use of qualitative reasoning before and after employing mathematical methods in solving physics problems [26,27].  Many strategies enforce adherence to the expert-like strategy by requiring students to perform each step via a marking rubric or worksheet. [1,20,21] The use of such a strategy and the requirement that students follow each step implicitly assumes that the students will, in some meaningful way, follow the instructions laid out in each step.  Given that these strategies are intended to induce students to make use of their conceptual knowledge at appropriate times during the problem-solving process, I examine students’ discussions to investigate whether these strategies are successful at doing so.  The fourth main research question in this dissertation is:  Research Question 4:  To what degree are structured problem-solving methods effective at promoting the use of conceptual and qualitative knowledge at the intended times in the solution process? 1.4 Dissertation Overview In subsequent chapters of this dissertation I first provide a thorough description of the changes made to the Physics 100 course, and then describe four studies that each address one or more of the research questions. These studies are introduced in the following subsections, and then described in detail in chapters 3 through 6.  In chapter 7 I discuss the results and implications of each study and suggest some future work. For reference, the major studies and their individual research questions are listed in Table 1 below.  The first column of Table 1 lists the main research questions.  Then in each subsequent column I list the ways that research question is addressed by the individual chapter.   8  Overall research questions Study of Changes in Students' Beliefs (Ch. 3) Exploratory Post-course interviews (Ch. 4) Structured RWC Interviews (Ch. 5) Framing and RWC in Collaborative Group Problem-Solving (Ch. 6) 1 Did the changes to physics 100 improve students' beliefs regarding the relevance of physics to the real world? Did the changes to physics 100 improve students' beliefs regarding the relevance of physics to the real world? 2 In what ways do students make connections between physics and the real world? In what ways do students see physics as related to the real world? In what ways do students make use of their real- world knowledge during collaborative group problem-solving? 3 What influences whether and how students make connections between physics and the real world?  What factors do students cite as affecting their belief in the relevance of physics to the real world? What features of scientific problems do students see as connected to the real world? How does students' framing affect whether and how they make use of their real-world knowledge during collaborative group problem-solving?  How does the structured problem solving strategy affect whether they make use of their real-world knowledge during collaborative group problem-solving? 4 To what degree are structured problem-solving methods effective at promoting the use of conceptual and qualitative knowledge at the intended times in the solution process? To what degree are structured problem- solving methods effective at promoting the use of conceptual and qualitative knowledge at the intended times in the solution process? Table 1:  Overall dissertation research questions and sub-questions by chapter. 1.4.1 Study of Changes in Students' Beliefs Chapter 3 describes the study conducted to examine the impact of the Physics 100 course changes on students’ beliefs.  In order to examine beliefs that affect how students make use of scientific knowledge in their everyday lives, I am particularly concerned with students’ perception of the relevance of physics to the world outside   9 their classroom.  To examine students’ perception of real-world connections in physics I use the CLASS survey of students’ attitudes and beliefs about physics [14], which was performed as a pre- and post-test prior to the transformation as well as in each year since.  The results from this survey from the years 2006 − 2009 are compared, and differences in instructional faculty, student demographics, and surveyed populations are incorporated into an ANOVA statistical analysis to explore the impact of the course changes. 1.4.2 Exploratory Post-Course Interviews Chapter 4 describes the Exploratory Post-Course Interviews, a semi-structured interview study that explores students’ perceptions of the course changes.  The interviews are structured around individual questions from the CLASS survey [14], all in the form of a statement that students are asked to agree or disagree with on a five point scale.  In the interviews students were asked to respond to various CLASS question prompts and explain their answers, as well as to give free-form comments on the course changes. This study addresses research questions 2 and 3 by examining the ways in which students see physics as related to the real world and the factors that they cite as affecting their belief in the relevance of physics to the real world.  Because the CLASS survey questions are used as the basis for the interviews, this study also serves to illuminate the students’ reasons for their CLASS survey responses within the specific context of the Physics 100 course. This study was conducted after the first year of the course transformation and informed the development of the course’s lecture materials, homework problems, and recitation problems for the next year.  Some of the students’ comments during this study helped to illuminate the difference between the faculty’s (and my) perception of physics that was relevant to the real world and the students’ perceptions of the same. This study illuminated several possibilities for major factors that influence students’ perception of the relevance of physics problems to the real world.  My new   10 understanding of these factors informed the Structured Real-World Connections interview study which examined these factors in a more systematic way. 1.4.3 Structured Real-World Connections Interviews Chapter 5 describes the Structured Real-World Connections Interview study. Based on the results of the prior study, I developed a diverse set of scientific and mathematical questions, each of which had one or more characteristics that I expected would lead students to either rate the question as very relevant or irrelevant to the real world.  For example, some questions were set in an everyday context such as a bus, others were set in an unusual context such as outer space, and others were completely abstract and had no context at all.  A wide variety of undergraduate students were recruited to give their judgment of the relevance of these problems in an interview format and to explain the reasons for their judgments.  The interview results were subsequently coded to identify the problem features that students cited as motivating their judgment of relevant or irrelevant to the real world.  These judgments were also subsequently coded and analyzed to identify which problem features were significantly correlated with high overall ratings of relevance. This study further examined research question number 3 and broadened it to include questions from science and mathematics disciplines other than physics. 1.4.4 Epistemological Framing and Real-World Connections in Structured Group Problem-Solving The results from the two interview studies above informed the development of the Physics 100 recitation problems, which were designed to promote the perception of relevance of physics to the real world and to encourage students to make use of their everyday knowledge within the recitation context.  In chapter 6 I describe the fourth study, which examines students’ response to these recitation problems to further illuminate the types of real-world connections that they make and to identify factors that promote or inhibit those connections. In this study I further examine research question 2 by analyzing students’ discourse while they engage with their normal weekly recitation problems.  I identify   11 moments when students make use of their real-world knowledge within this context, and identify a taxonomy of different types of these Real-World Connections (RWC).  I make use of the Resources and Framing theoretical framework to allow me to clearly define when a Real-World Connection occurs [28,29].  Briefly, this framework describes knowledge as consisting of fine-grained interconnected resources. Resources may be declarative, procedural, epistemological, or intuitive [30].  Any facts or principles that we make use of in order to reason may be considered to be resources, including ideas about knowledge and learning.  The use (or activation) of a particular resource may be context-sensitive and subject to different activation in different contexts.  For the purposes of this study, I define the use of real-world resources within the recitation context as a Real-World Connection. I argue that these Real-World Connections enable students to integrate their physics knowledge with their informal knowledge and also have the potential to influence students’ perceptions of the relevance of physics to the real world. One of the factors that I examine and correlate with students’ Real-World Connections is their epistemological framing.  An epistemological frame is a person’s implicit sense of the nature of the activity they are engaged in, and corresponds to a network of epistemological resources that tend to co-occur in a stable fashion.  The resources that are activated in a particular epistemological frame may include judgments of which knowledge is valid, what constitutes progress, and how that progress can be achieved within that activity.  In this way, frames can affect students’ approach to learning and affect which other resources they bring to bear in a particular situation [31].  In this study I correlate moments when students make a Real-World Connection with their epistemological framing to demonstrate how framing regulates the use of their everyday resources. I also examine how the structured problem-solving method used in the Physics 100 recitations correlates with students’ Real-World Connections and their epistemological framing.  This allows me to investigate research question 3 to determine how the strategy influences their Real-World Connections and research   12 question 4 to determine how the structured problem-solving strategy influences students’ use of conceptual and qualitative knowledge during problem-solving.   13 2 DEVELOPMENT OF PHYSICS 100 Because my research was largely conducted in and around the transformation of UBC’s Physics 100 course in 2007, this chapter provides background on that course and the descriptions of the changes. 2.1 Background of Physics 100 Physics 100 at the University of British Columbia is an algebra-based introductory physics course offered to those students who did not take senior high school physics. Prior to the course changes outlined in this thesis, the Physics 100 syllabus and format was similar to many North American introductory physics courses. The course consisted of 3 hours of weekly lectures supplemented with bi-weekly alternating 3-hour laboratory sessions and 2-hour optional recitation sessions where students worked in groups on practice problems. The course followed a common sequence of topics in mechanics, DC circuits, and geometrical optics. To improve student engagement, the faculty used an electronic response system (also known as clickers) to periodically ask short questions during lectures. Because Physics 100 is required for many of UBC’s Arts and Science programs the student population is very diverse. Approximately 60% of the students are B.Sc. students, but the vast majority of them are not intending to major in physics and are required to take only one physics course in addition to Physics 100. The remainder of the students are human kinetics, food and nutrition science, forestry, or arts students, and Physics 100 is likely the only physics course they will take in their undergraduate program. 2.2 Motivation for Changes After teaching the course for several years, the instructors of Physics 100 were concerned that students were not being well-served by the course content and format. This course presented physics content in ways that might be useful for further study in physics but few, if any of its students would be pursuing further studies in physics. The overriding question for developing a new version of this course was: other than a fulfillment of requirements for a degree program, what use can a physics course be to a student who is not principally interested in physics?   14 This perspective on making the course useful to the students underlies several complementary motivations for making changes to the course.  These are discussed in the sections below. 2.2.1 Students’ Understanding of Societally Relevant Issues Because the students in Physics 100 would likely study very little physics in their university careers, the instructors hoped to teach them something that would be useful outside physics class.  To that end they considered both the applications of physics in other courses and the applications of physics to real-world problems and issues.  The key real-world issues the instructors wanted to address were energy conservation and climate change. An understanding of issues in climate change and energy conservation are crucial to the success and prosperity of society but the feeling of the instructors was that the public (including many journalists) may not understand them very well. Worse, it seemed that many people adopted an attitude of either taking scientific statements in the media at face value or writing off all science as ‘just theories’; taking the presence of scientific debate as an indicator that nothing in these issues was settled or worthy of action. The instructors hoped that including a discussion of these issues as well as demonstrating techniques for critical analysis of arguments presented in the media might help to raise their students’ understanding of these issues as well as their ability to meaningfully engage with these issues in their everyday lives.  They intended that this course would contribute to the students’ scientific literacy, which for them meant that they would understand enough about science content and the nature of professional science that they would be willing and able to use their own knowledge to evaluate scientific messages in the media and take action to engage with socioscientific issues. 2.2.2 Retention The instructors were also concerned about students’ poor retention of facts and concepts learned in physics class. It had become clear from their prior experience as   15 well as academic research that the learning strategies employed by students, which often involved last-minute cramming and “memorization” of textbooks, resulted in shallow and temporary knowledge [32,33]. This was particularly true of students in Physics 100 who would take only one or two physics courses, and would have little opportunity to re-learn and deepen their understanding of physics concepts in later courses.  The Physics 100 instructors were concerned that unless they changed their students’ approach to learning in the course their students would literally forget everything they learned. 2.2.3 Perception of Relevance of Physics The instructors came to believe that a key piece of this problem of retention was the issue of relevance: the Physics 100 students did not see the physics presented as being relevant to anything outside of the physics classroom, and therefore had no motivation to learn it in a deep way. The head instructor’s feeling was “Even the best student will expel [physics] knowledge if they don’t see it as being relevant”. 2.2.4 Decline in CLASS Survey Scores The instructors’ concern about the students’ perception of the relevance of physics was also highlighted by research on student attitudes and beliefs about physics.  Several researchers had examined students’ attitudes and expectations on several areas relevant to learning in physics, such as students’ perception of knowledge and learning, relevance to the real world, and the nature of problem solving in physics [10,13]. In the 2006 Physics 100 course, the Colorado Learning and Attitudes towards Science Survey (CLASS) was administered as a means of measuring how students’ attitudes and expectations towards physics had changed after taking the course  [14]. The CLASS survey uses 42 statements about physics and learning with which the students are asked to agree or disagree on a 5-point Likert scale. Each statement has a favourable or unfavourable response, as defined by the response that is consistently given by physics professors and other expert physicists.  As is often the case in traditional introductory physics courses, the average CLASS score of Physics 100 students in 2006 became less favourable over the course of instruction [34]. This   16 suggested that the course was actually having a deleterious impact on student attitudes toward physics, reinforcing undesirable attitudes and assumptions about the nature of physics and learning. This negative impact in Physics 100 was especially concerning because the instructors felt that these negative opinions would be many students’ last impression of Physics, and could colour the way their students perceive physicists and scientific information for the rest of their lives. 2.2.5 Real-World Problem-Solving Skills The instructors expected that including material on energy conservation and climate change would provide students a better understanding of those issues, but they felt that physics had more to offer than specific knowledge about particular issues.  The instructors felt that this course represented an opportunity for students to learn how to model complex, novel problems and estimate solutions using quantitative problem-solving methods. These modeling and estimation skills are one of the hallmarks of expert problem- solving, and the instructors knew from their own experience that they can be very useful in a broad variety circumstances, both academic and everyday.  They felt that traditional physics courses over-emphasized purely mathematical aspects of problem solving, without addressing how mathematical representations are created in the first place.  In order to enable students to learn flexible problem-solving skills, the instructors felt it was important to demonstrate how simplified models were created from realistic situations and how the results of calculations can be checked against our expectations of the real situation.   They hoped that highlighting the diverse practices needed for solving real-world problems would help students develop a more complete set of skills for tackling novel complex problems. 2.3 Goals of Changes After much discussion, the Physics 100 instructors settled on three major goals for their course.  They wanted to:   17 1. educate their students about socially relevant issues 2. enable them to learn how to apply physics to other scientific issues in the public sphere 3. encourage students to see physics as relevant to themselves and to their lives The instructional team was not aware of any prior pedagogical interventions focused on use of connections to everyday problems and contexts as a tool to promote student engagement and learning in university-level introductory physics.  They felt that their goals were attainable within the introductory physics setting, and set about developing a curriculum and pedagogy that would support these goals. 2.4 Changes to Physics 100 In order to meet their goals for this course I worked with the Physics 100 instructors to make changes in a wide variety of aspects of the course.  In an endeavour to offer physics education that would be useful to students outside of the classroom we added several physics topics relevant to thermal physics, climate change and energy production to the course content, reworked many examples and problems to put them in a more everyday context, increased the course’s focus on problem-solving skills by adopting a prescribed problem-solving strategy in lecture and recitations, and implemented a research project. These changes are described in detail in the following sections.  These descriptions are not intended to be comprehensive, but rather to give the reader a sense of the course which was used as a setting for my research.  Special attention is paid to the changes to the course’s recitations, which was the setting for one of the main studies in this dissertation. 2.4.1 Everyday Context In order to enable students to apply physics knowledge in everyday situations and to demonstrate the relevance of physics to the real world, the faculty felt that it would be crucial to situate the physics in the context of the students’ everyday world.   I worked along with the course faculty to find specific example problems, diagrams, and instructional methods that would present this new course syllabus in terms of the   18 students’ everyday experience.  As much as possible, every example in the lectures, every homework question, and every recitation question was set in a context that students could plausibly have encountered in their lives. This task proved to be quite challenging. Traditional physics instruction is often engaged in the business of reduction: each physical effect is discussed and demonstrated in absence of other effects, and far from the complex reality of everyday situations. Unfortunately examples that are rooted in the everyday world of the students and cleanly demonstrate the action of a single physics principle are rare: most everyday situations involve a combination of several influences which makes any discussion of the physics quite complicated, especially for novice learners. In order to address this problem we identified several “models” that we wanted to teach: example calculations describing situations that were relevant and useful which could act as anchoring ideas to help motivate the development of the constituent physics.  Examples of these models include:  the energy flows and temperature in a home, the energy balance and temperature of the earth, the fuel consumption of a car or bus, and the efficiency of a system of electricity transmission.  Working with these complex models meant that we needed to foreground the importance of simplifying assumptions in rendering complex everyday systems tractable. While the emphasis on real world relevance of course content is not new in the global sphere of education, this commitment to present an entire introductory university physics course in an everyday context was ground-breaking. However, as discussed below in section 4.5, it became clear after the first year of transformation that the faculty’s perception of what was ‘relevant’ and ‘everyday’ did not always match the students’ perception. 2.4.2 Course Content The idea of conservation of energy was used as a touchstone throughout the course, making connections between the various applications to thermal physics, mechanics, and electricity.  Energy was expanded from a single unit in mechanics to a unifying theme for the entire course. A discussion of fuel efficiency and energy   19 consumption in transportation was added to the material on mechanics, and the unit on electricity was contextualized in discussions of household energy consumption. Because of the prevalence of energy in the curriculum, the faculty decided to try to construct a lesson sequence that started with a discussion of energy.  This decision was not easy; it was a challenge to find a new way of introducing this material in a way that deviated from the traditional method of introducing energy only after kinematics and dynamics had been introduced. On one hand, we felt that observable properties of objects such as velocity and height would be easier to explain to students, so from this perspective it seems sensible to use the more traditional approach of discussing the more abstract concept of energy only after kinematics and dynamics have been thoroughly explored. However, I suggested that energy as a scalar quantity might be mathematically easier to deal with at the beginning of the course than the vector trio of position, momentum, and velocity, and might therefore be more accessible to students at the beginning of the course. In the end we all agreed that the course should be “about energy”, and so the faculty agreed to try introducing energy at the beginning of the course and referring to this concept throughout. It is clear that the faculty believed that learning about energy was relevant to their students. In support of this principle and the goal of educating the students about socially relevant issues that they could connect to their everyday lives several topics were added and some were removed from the course.  These are discussed below 2.4.2.1 Thermal Physics and Climate Change One of the key topics we wanted to address was the physics of climate change.  One of the other professors on the development team had already been teaching the physics of home heating, a system that can be modeled using many of the same physics concepts as those used to model the Earth’s thermal equilibrium. The problem of understanding home heating seemed like the perfect context to motivate the development of the thermal physics required for climate change.   20 For this reason we added a quantitative treatment of conduction and radiation and a qualitative discussion of convection to the course.  These concepts were used to develop a model of thermal equilibrium in home heating and subsequently used to discuss the radiative energy balance of the earth and its relevance to climate change. The faculty agreed that climate change was a worthwhile topic to teach, but were concerned about the challenge of interpreting this very complex system for an introductory audience. However we found an introductory geoscience text that offered several perspectives on how simple models of climate change could be used to illustrate important characteristics of the Earth’s energy system [35].  These models neglected all dynamics of the atmosphere and treated it simply as a single layer with different absorption for different wavelengths of light. The instructors agreed that this model would strike a balance between simplicity and complexity, and would allow us to discuss important features of the Earth’s energy balance and highlight the critical role of the absorption of thermal energy by the atmosphere in determining the surface temperature of the Earth. 2.4.2.2  Air Resistance and Energy in Transportation In order to allow for discussion of fuel consumption in transportation the basic physics of air resistance was added to the course.  This, in combination with the course’s existing treatment of friction, supported the development of a model of the energy required for a car to travel at a constant rate which supplemented the existing curricular topic of the energy required to accelerate up to speed and to climb hills.  Taken together, these principles allowed us to develop a simple model of fuel consumption in transportation and enabled comparisons between different modes of transportation (e.g. cars, buses, and trains) as well as between different engine types (e.g. internal combustion and electric vehicles).   21 2.4.2.3 Chemical and Metabolic Energy In support of the goal of discussing the physics of everyday life, a discussion of chemical and metabolic energy was added to the course.  We developed materials on the energy content of food and the energy consumption of various physical activities.  This material ties into the discussion of thermal energy, as the occupants of a house help to heat its interior. Introducing basic concepts of chemical energy also allowed us to discuss the fuel consumption of various forms of transportation and allowed us to draw analogies between a car’s consumption of fuel and a person’s consumption of food. 2.4.2.4 Power in Circuits The course’s unit on DC circuits was re-factored to support a discussion of household electricity and power consumption.  This meant introducing the concept of AC circuits, but the quantitative treatment was limited to RMS values.  The concept of wires with finite resistance was added so that energy losses in electrical transmission could be quantitatively analyzed.  The emphasis of this unit was in understanding power consumption in household circumstances, so much of the discussion centered on simple parallel circuits such as those found in household wiring.  Detailed discussions of current and voltage in combined series and parallel circuits were eliminated. 2.4.2.5 Eliminated Topics In order to make room for the added topics, we needed to eliminate some topics from the curriculum.  The first major topic that was removed was conservation of momentum.  The best real-world examples of this principle are collisions, but we felt that this application is a rare event, often removed from students’ typical lives and is principally of use to technical professionals such as accident investigators and, of course, physicists.  Knowledge of momentum won’t help prevent a collision, and knowing about it afterwards won’t help you deal with one, so we felt that this topic had limited use for students that were   22 not intending to continue in physics.  To make room for more applicable everyday concepts it was deleted. Vector analysis was another topic that faculty felt was of limited value to this student population. For physics majors, the mathematical and conceptual tools of vector decomposition are essential, but the instructors’ experience had been that students in the Physics 100 course tended to regard these aspects of physics as purely formulaic and meaningless exercises.  Therefore, vector analysis was cut from the mechanics curriculum along with all discussion of two-dimensional motion such as circular motion, cars on inclined planes, and projectile motion.  While these topics certainly have real-world applications, we felt that the additional computational challenge was not warranted, and we could teach other topics that would be more valuable to students that were not as challenging. As mentioned above, the discussion of circuits was significantly simplified to focus on energy transmission and consumption, and detailed discussion of current and voltage in complex DC circuits was eliminated. Finally, the course’s two-week unit on geometrical optics was eliminated. We could not think of many compelling ways for a non-physicist to make use of simple geometrical optics, and so this topic was eliminated. 2.4.3 Problem-Solving Skills The course transformation was significantly shaped by our consideration of what students would retain from this course. Research on student retention of concepts from introductory physics courses supported the faculty members’ experience that students retained little declarative knowledge from their prior courses [36]. However the faculty believed that formal education can still be useful after one has forgotten the facts and formulae. They felt that often a way of approaching problems (such as the practice of cutting them down into smaller pieces) can help people interpret new situations. This led us to try to highlight things that we felt could be remembered:   23 problem solving approaches that might allow our students to address real world problems where physics might be fruitfully applied. This perspective guided us to try to emphasize opportunities for students to learn how to apply physics in real world situations, rather than to simply amass declarative knowledge of physics content. The intent was not to do away with learning of declarative knowledge altogether, but to emphasize that the key skills of recognizing which knowledge is applicable and how to apply this knowledge. To support the development of students’ problem-solving skills, we changed the weekly optional problem-solving sessions to mandatory group work on ‘context-rich’ problems based on those developed by Heller and Hollabaugh [1].  We also added a group research project to the course, where students develop a model and conduct a calculation in order to answer a question or make a decision about a real-world problem.  Finally, in keeping with the recommendations of several physics education researchers (e.g. [20,21,37]), we adopted a structured problem-solving method that was used in the recitations and in lecture examples to demonstrate the process of applying physics in a novel situation.  These changes are described in more detail below. 2.4.3.1 Modeling and Assumptions Modeling is at the heart of physics analysis, and assumptions of rigidity, linearity, and frictionlessness are common in introductory physics courses in order to render problems tractable.  Models are often communicated through simplified drawings and terse lists of assumptions and initial conditions. However students are rarely exposed to the process of translating a complex real-world situation into a simplified model, and are often presented with only the abstracted result.  This restricts their ability to make use of physics when confronted with a realistic situation, and contributes to their perception that physics is disconnected from reality.  A student that only learns physics in the context of boxes on planes may never think of it when they are trying to move furniture across a carpet.  Exposing the process of abstraction and   24 modeling is necessary in order to demonstrate the intimate connection between physics and the real world. In order to highlight the importance of modeling and assumptions in physics we included some explicit discussion of these issues in the course. This process of making assumptions to simplify everyday situations was discussed in some detail throughout the course and was reinforced in the recitation sessions which required the students to make assumptions as a part of developing their own models. By discussing these techniques explicitly, we intended to address some of the students’ customary discomfort with making sweeping approximations. Another strategy for dealing with students’ concern about making extreme simplifications such as ‘frictionless’ or ‘rigid’ was to search for real world examples where these could be justified via commonsense reasoning. This task was quite difficult, but we did find a few examples of situations where we believed students would accept such simplifications as being natural. 2.4.4 Structured Problem-Solving Prior research in problem-solving has shown that novice problem solvers typically use a narrow suite of problem-solving strategies and unlike experts, frequently do little planning, visualization, or qualitative description prior to attempting a solution [27,38].  Since they are often skilled at pattern matching and algebraic manipulation of equations, these “plug and chug” techniques can allow students to score well on quantitative problems without an understanding of the qualitative physics [39]. To support the development of students’ problem-solving skills a structured problem-solving method was introduced in the Physics 100 lectures and recitations. Problem solving was presented to the students as a series of steps which scaffold the process of interpreting and structuring a problem, applying the appropriate physics principles to obtain a solution, and interpreting the solution.  This method was used by the instructors during class and was used to scaffold the recitation problems   25 themselves with a series of written prompts and corresponding blank space for students’ work.  Constraining novices to use more expert-like solution processes is a widely-used practice intended to foster their development of expert-like problem- solving skills.  Among the pedagogies that advocate prescribed problem-solving methods are Heller and Hollabaugh’s context-rich problems [1], Teodorescu’s ACCESS protocol [20], and Van Heuvelen’s Active Learning Problem Sheets [21]. The specific steps used in Physics 100 were based on Heller et al.’s Cooperative Group Problem-Solving (CGPS) strategy but were modified over the first three years of implementation in Physics 100 as the instructors adapted to feedback from the TAs and researchers observing the course.  The problem-solving strategy described below was the strategy we settled on in 2009, the third year of implementation.   Each step is described below, and similarities to existing problem-solving schemes are noted. 2.4.4.1 Step 1: Interpret the Problem Students often dive into calculation on a problem before they have attempted to interpret the situation, visualize what is happening, or identify the goal of the problem.  This step directs students to make a quick sketch of the problem to aid in forming a mental picture and narrative of the physical process under discussion.  This initial step mirrors the Heller’s CGPS step “Visualize the Problem”.  Indeed, the context-rich problems used both in Minnesota and at UBC eschew illustration precisely to force students to engage in visualization and translation to visual representations. This step also directs the students to identify the goal of the problem, something which is not always explicitly stated.  As in the Minnesota context- rich problems, the task of identifying what must be calculated and/or compared is something we want the students to learn how to do.  Explicit statement of the problem goal is also explicitly prompted by the first step of the GWU ACCESS protocol which directs students to  “Assess the problem”.  Curiously, identification of the problem’s quantitative goal is NOT explicitly prompted by the CGPS strategy, although it is certainly necessary for students to interpret the   26 goal of the problem in order for them to “Plan a solution” which is the third step of the CGPS strategy. 2.4.4.2 Step 2: Identify Relevant Physics A key result of expert-novice problem solving research is that experts categorize physics problems based on their “deep structure” whereas novices focus on the “surface features” [26].  In order to encourage students to focus on the deep physics of the problems, this step directs them to explicitly state the physics concepts and principles that are relevant to the physical process being described. This step is mirrored in the “Assess the problem” step of Teodorescu’s ACCESS strategy.  While it is not explicitly directed in Heller’s CGPS strategy worksheets, selection of the relevant physics is a key feature of the Problem Solving Rubric used to mark the context-rich problems in Minnesota [40] and so is certainly implicitly emphasized and reinforced. 2.4.4.3 Step 3 & 4: Create a Physics Model In order to reinforce the connections between physics and the real world students are asked to explicitly lay out their translation from a description of a realistic situation to a mathematical model.  They are explicitly prompted to create a diagram, as well as to identify which assumptions are needed in order to proceed with their solution. By explicitly prompting (and emphasizing via the marking rubric) both the drawing and the assumptions, we hoped to reinforce the importance of recognizing the limits and capabilities of the simple equations used in introductory physics, as well as encourage students to see these problems as mathematically corresponding to physical situations.  We also intended to draw students’ attention to the key position that model construction holds in solving complex problems.   27 Several other problem-solving schemes also explicitly require this type of preparation for solution. This step is mirrored by the ACCESS protocol’s “Create a Drawing” step, which explicitly prompts for sketches or specific graphical representations such as a free-body diagram or chronological diagram. Similarly, the Minnesota CGPS method encourages students to “represent the problem in physics terms” and explicitly prompts creation of graphical representations such as vector or free-body diagrams.   These aspects are assessed by Docktor’s problem solving rubric under the headings Useful Description and Specific Application of Physics. The SCALE-UP GOAL problem-solving method (and Polya’s after which GOAL is modeled) also encourages students to make a drawing and summarize relevant information [41,42].  However we were concerned that these strategies present the modeling step as only summarizing the quantitative information necessary for a calculation, rather than constituting an important qualitative reasoning tool in its own right.  Expert physicists commonly use diagrams to monitor and check their solutions-in-progress, and we hoped to demonstrate the value of these diagrams and to have students to practice developing them. These strategies also do not emphasize the importance of making simplifying assumptions, something that we felt was essential for tackling real-world problems. The information requested in these steps together with the physics principles used in the previous step corresponds to Hestenes’ description of a mathematical model [43,44].  While this use of the term “model” is not as broad as that implied by the Modeling Instruction curriculum, we similarly expect to send the message that even though physics calculations may contain a number of assumptions or imperfections, they nevertheless rest on generalizable quantifiable principles that can give useful information about complex real-world situations.   28 2.4.4.4 Step 5: Solve the Problem This step is the most commonly cited in each of the structured problem- solving methods in the PER literature, and is often the step that novice students try to start at. In this step we direct the students to make use of the information in their model and the relevant physics they had previously identified in order to solve for the goal variable.  Beichner’s GOAL strategy similarly directs students to analyze the problem by selecting relevant equations to solve for the unknown variable. 2.4.4.5 Step 6:  Error-Checking and Sensemaking In this step the students are prompted to reconsider their answer in terms of their everyday knowledge and to compare it to any quantitative benchmarks that they may happen to know. Novices are often over-confident in their answers and decline to scrutinize or question them.  This step explicitly directs them to use their commonsense to evaluate the answer, as well as to connect it back to the purpose or consequences of the original question.  We expected that this reflection would help students to see the real-world connections in physics. We deliberately used the term error-checking AND sensemaking here to signal to students that these are two different activities.  Students focused on getting the right answer will often stop at error-checking.  However, a solution free of algebraic errors can still be wrong and lead to the wrong conclusions if it is based on faulty assumptions.   We tried to highlight sensemaking as the process of double-checking the meaning of the real-world context and the mathematical results. A similar direction instructing students to reflect on the reasonableness of their result is a part of nearly every published problem-solving strategy. However, these strategies vary in their focus, with some placing much greater emphasis on broader metacognitive questions about the students’ strategies and   29 learning during the solution.  The Minnesota CGPS strategy directs students to check their solution with their prompts for completeness, correct units, and reasonable magnitudes, procedures which focus primarily on the problem itself. The ACCESS protocol directs students to “Scrutinize your results” with an additional direction to “Compare the situation presented in the problem with a real-world situation,”   similar to our instruction for students to make use of real-world benchmarks. Polya and the GOAL strategy both instruct students to “look back” over their solution to see if it meets their expectations and “makes sense”.  The GOAL strategy also asks more far-reaching questions, prompting students to consider this problem in the context of others in the course, to try to figure out why it was assigned, and to examine what was learned from doing it.  Our strategy lacks this explicit metacognitive direction. 2.4.4.6 Complete Physics 100 Problem Solving Strategy The complete Physics 100 problem-solving strategy is listed below.  Table 2 lists the titles of the problem-solving steps that were used in lecture and recitation.  Table 3 lists the full prompts that were printed on the recitation worksheets.         Table 2: Summary of the structured problem-solving strategy used in Physics 100.  These are the titles of the problem-solving steps that were used in lecture and recitation. Step Step Name 1 Interpret the Problem 2 Identify Relevant Physics 3 Model:  Define Assumptions and Relationships 4 Model:  Construct a Diagram 5 Solve the Problem 6 Error-Checking and Sensemaking   30  Table 3: Detail of the structured problem-solving strategy used in Physics 100.  These are the prompts that were written on the recitation worksheets completed by the students.  1. Interpret the problem • Carefully read and visualize the events described in the problem • If necessary, sketch a picture to clarify sizes, directions and spatial relationships • Clearly state the GOAL of the problem:  what do you need to calculate and/or compare  2. Identify the relevant Physics Concepts • List briefly the major physical principles that are relevant to this situation.  (e.g. conservation of energy, Newton’s second law, 1-D kinematics with constant acceleration, etc.)  3. Model:   Define Physics Assumptions and Relationships • Using words and formulas as appropriate, interpret how the situation affects the physics variables defined in step 3b.  Be sure to state all: • Limitations or constraints on the physical variables (e.g. v < 0) • Relationships between the physical variables  (e.g. a1 = a2) • Simplifying assumptions (i.e. friction negligible, massless rope, constant acceleration etc.) • Initial conditions (i.e. Vi = 0, ai = -g)  4. Model:  Make a Diagram and Summarize the Relevant Information • Write down a description that summarizes all of the relevant information in the problem statement in a clear way.  This could include pictures, equations, or descriptions, or any of the following, where appropriate: a. A statement of known and unknown quantities, with appropriate symbols defined b. Any intuitions or expectations about the answer.  (e.g. “because of the situation in this problem, I would expect the speed of car B to be only a little bit faster than that of car A”) • A specific physics diagram (free-body diagram, energy bar chart, motion diagram etc.) • A coordinate system that specifies the reference and direction of measurement for any spatial variables.  (e.g. “x = 0 is the tabletop, positive x is down” or equivalent symbols  5. Solve the Problem • Show all calculations  6. Check Your Answer  (Error Checking / Sensemaking) • Demonstrate that your result has the correct units • Compare results to known benchmarks • If necessary, perform additional calculations to check your answer is sensible   31 The detailed prompts depicted in Table 3 have several elaborating statements below them explaining the required information for each step.  For the first context-rich recitation in Physics 100, these prompts appeared as depicted, including the full elaboration.  After the students had completed two context-rich recitations the italicized text was removed and only the titles of each step remained.  Other than this change, the text of each prompt was the same from one recitation to the next, although the order of steps 3 and 4 was sometimes varied depending on the situation. 2.4.4.7 On the Lack of a Planning Step The CGPS and ACCESS strategies explicitly direct students to plan their solution, and then the Solve step is framed in terms of executing their plan. These recommendations are based on research suggesting that novices are less likely to explicitly plan a problem solution.  An explicit step directing students to plan their solution was originally included as an explicit part of our problem- solving strategy, but was cut after the first two years of implementation. The main reason for cutting this step was lack of buy-in from the students. Informal observations of students in the recitations corroborated TAs’ reports that students would typically solve the problem and then return to fill in the plan step later, thereby evading any benefit gained from pre-planning.  Despite repeated encouragements, most students never authentically attempted to plan their solution There are several possible explanations for this.  One important possibility is students’ motivation to maximize their grade.  Students often feel time- pressured during the recitation sessions, and despite the fact that this step was explicitly encouraged and rewarded with marks, students may have felt that the most efficient way to get as many marks as possible is to solve the problem before developing a plan.  This “solve first, plan later” strategy enabled students to start with the physics strategies that they were more familiar with from high school:  pattern-matching equations and variables and working through an algebraic solution.   32 In fact this may have been the optimal strategy for our recitation problems. It was very difficult to build problems that were difficult enough that they required forethought and planning without making them so difficult that some students would be unable to complete them.  Another concern was that the problems would seem so difficult on first glance that students would feel that they didn’t know how to get started on the problems.  Despite the fact that the structured problem-solving method is designed to help students work through problems they have no immediate idea how to solve, we were concerned that students would become daunted and refuse to meaningfully engage in a problem that they felt is too hard for them.  In the interest of making the problems accessible to the extremely diverse student population of Physics 100, we may have erred on the side of making the problems too easy, which would make it possible to solve them without a planning step. Another possible reason for students’ failure to adopt the planning step of our problem-solving strategy was insufficient support from the course TAs. Many TAs expressed skepticism about this step, and weren’t trained sufficiently on the motivation for including this step in the recitations.  Without a firm understanding of why it was included, TAs could only justify this step to students as a rote part of the required process.  This lack of a clear vision of how this step is helpful to the overall process made it just an arbitrary hoop to jump through and students treated it accordingly by performing it in haste right at the end of the recitation session. Finally, observations in lecture showed that course faculty rarely demonstrated an explicit planning step before solving example problems, further reinforcing the idea that this step was unnecessary.  Therefore, after two years it was dropped from our problem-solving method. 2.4.4.8 Scoring The recitation grades were assessed with a set of rubrics that assessed students’ performance in each of the problem-solving steps.  Rubrics are descriptive scoring schemes that contain a description of different levels of   33 performance on each of the steps, including the “perfect” level.  These rubrics aid the TAs in grading the recitation problems and are also designed aid the students in self-assessing their performance and interpreting the feedback given by TAs.  Formative feedback has been demonstrated as effective at producing learning gains [45,46]. Our problem-solving rubric was based directly on the Docktor rubric used in Minnesota’s CGPS [40].  However, a few changes were made in order to enforce students’ participation in each step of the problem-solving strategy. Firstly, we gave students equal points for each step of the problem-solving strategy, clearly sending the message that every step of the process is important. Also, because we require students to fill something in for each step of the problem-solving strategy we always scored each step individually rather than allowing the students to demonstrate implicitly that they had done a step correctly by their work on other steps (as Docktor’s rubric does).   The course instructors also felt that this one-to-one correspondence between the problem solving steps and assessment rubrics would be easier for the students and TAs to understand and therefore lead to more meaningful feedback and more consistent marking among the TAs. As with many other aspects of the course, this rubric evolved over the first three years of implementation based on student and TA feedback.  The figure below shows our rubric from the third year of implementation.    34 Table 4:  Marking rubric used for Physics 100 recitation problems. 2.4.5 Changes to Recitations The following sections describe the changes made to the Physics 100 recitations as a part of the course transformation.  Because data-gathering for my major research project that was conducted in the recitation context occurred in 2009 (the third year Rubric Mark 5 4 3 2 1 0 Similar to a grade  of >80% ~75% ~50% ~30% <30%  Appropriate, Useful, and Complete Appropriate, Useful, and Complete with minor omissions Partly erroneous, inappropriate, or incomplete Mostly erroneous, inappropriate, or incomplete Completely erroneous or inappropriate Missing 1.  Interpret the Problem Problem summary and goal statement is appropriate, useful, and complete Problem summary and goal contains minor omissions or errors Parts of the problem summary and goal are missing or erroneous Most of the problem summary and goal are missing or erroneous All of the problem summary and goal is erroneous No attempt is made to summarize and identify the problem goal 2.   Identify Relevant Physics Concepts The relevant physics is identified according to correct physics principles and concepts The relevant physics is identified according to correct physics principles and concepts, with minor omissions or errors The relevant physics is identified correctly but according to specific formulas or concepts rather than general principles) Identification of the relevant physics is mostly missing or inappropriate The relevant physics is assessed incorrectly, or is entirely described in terms of surface features of the problem No attempt is made to identify the relevant physics 3.  Model: Define Assumptions and Relationships All defined assumptions and relationships are appropriate, useful, and complete Assumptions and relationships have minor omissions, errors, or irrelevant statements Parts of the assumptions and relationships are missing, irrelevant, or erroneous. Most of the assumptions and relationships are missing, irrelevant, or erroneous. All of the assumptions and relationships are missing, irrelevant, or erroneous. No relevant assumptions or relationships are listed 4.   Model: Diagram and Relevant Information The physics model is useful, complete, and appropriate. The physics model has only minor omissions or errors The physics model has major omissions or errors The physics model is mostly erroneous or inappropriate The physics model is completely inappropriate or erroneous No attempt is made to construct a physics model 5.   Solve the Problem The problem solution is appropriate and complete, with symbolic equations until the last step The problem solution is appropriate and complete, with symbolic equations until the last step, with minor omissions or errors The problem solution contains some good equations, but some equations are missing or inappropriate The solution contains mostly wrong equations or solely numerical equations. The solution contains completely wrong equations or solely numerical equations.  No attempt to solve the problem symbolically is made No solution is attempted 6.   Check your Answer The check of answer units and magnitude is appropriate and complete.  Answer is compared to known values and units are explicitly checked Check of answer units and magnitudes contains minor omissions or errors. Check of answer units and magnitudes contains major omissions or errors. Either the explicit check of units or comparison to known values is missing Check of answer units and magnitudes is mostly incorrect or missing. The check of answer is vague or irrelevant. E.g. "seems reasonable" No check of the answer is attempted   35 of the transformation) these descriptions focus on the version of the recitations in that year. During that year there were nine recitations: Tutorials 1- 5 were workshops focused on teaching students individual problem- solving skills. See section 2.5.4.3 below for a description of these. Tutorial 6 was a Jeopardy Question [47], which is not discussed in detail in this document. Tutorials 7, 8, and 9 were context-rich recitations.  See section 2.4.5.1 for a description, and chapter 6 for the research conducted on these recitations. All of the problems used in the 2007 and 2009 year of Physics 100 are reproduced in Appendix A. 2.4.5.1 Context-Rich Recitations The instructors felt that standard end-of-chapter textbook problems helped the students gain facility with algebraic manipulation and pattern matching, but they often failed to advance students’ understandings of the “real physics” of the problems that they were doing.  This finding has been echoed by research showing that ability to solve standard problems does not necessarily equate to conceptual understanding [39].  In order to meet their goal of teaching students how to apply physics knowledge to real-world situations other types of learning tasks would be required.  context-rich problems were chosen for the course based on their claim to improve student problem-solving on novel problems [1]. These problems are organized around quantitative calculations, but are set in real-world contexts and often contain rich description and extraneous information, much like a real-world setting.  They may also require students to make assumptions about quantities or relationships based on their knowledge of the problem context.  In emphasizing the role of assumptions in application of   36 physics, we agree with Fortus [48], who has researched and argues for the need to teach students to make assumptions in introductory college physics. Overall, the main reasons we chose context-rich problems to be the basis for the Physics 100 recitations are: 1. They fit with our goal of using realistic contexts to demonstrate situations where physics is relevant to the real world. 2. We felt they would give students the opportunity to develop the broad variety of skills required to model a realistic situation and interpret the situation. 3. This style of problem makes good use of the TAs.  Having students solve challenging problems during sessions where the TAs are present creates opportunities for the students to learn from the TAs. 4. Using challenging problems for group work has been shown to increase students’ interactions with each other, creating more opportunities for students to learn from each other. 2.4.5.2 Group Structure and Roles As suggested, these problems were given to the students in groups.  Course logistics prevented the use of the recommended three-student groups, so students were put into groups of four. We implemented Heller et al.’s suggested system of rotating group roles.  In this system, students are assigned a particular role each week, such as Group Manager, responsible for proposing ideas and keeping the group moving forward; Skeptic, responsible for challenging the Group Manager’s ideas to make sure they are correct; or Recorder, who helps to facilitate group consensus and records the results.  Because we were working with 4 students per group, we created a fourth role, the Explainer, who was responsible for explaining new ideas to other team members and to the TA.  At the beginning of term we gave students an explanation of the importance and function of each   37 role.  As recommended by Beichner and Saul [49], we endeavored to ensure that the groups were heterogeneous in ability (based on previous grades in physics and math) and that females were not in the minority of a particular group.  We also shuffled the groups several times throughout the term, reassigning group members randomly while still following the above guidelines. In the first year of implementation students expressed a significant amount of negative feedback about these externally-imposed group restrictions.  The changing of groups in particular was met with significant resistance.  Because the assigned groups were also used for the end-of-term Research Project (see section 2.4.7 below) students also complained of difficulties in coordinating out-of-class times to meet with their group to work on this project.  Because UBC serves a geographically diverse student population with more than 50% of students commuting 30 minutes or more to campus, this was a significant challenge for students in scheduling meeting times and locations [50]. In addition, TA feedback and observations of students during the recitation sessions suggested that students did not take the group roles seriously, and would often just write down their roles that were required that week at the end of the recitation, having paid little or no attention to their assigned role during the recitation itself. In order to address these problems, we experimented with different versions of the group structuring rules.  The major study of student behavior during the recitations was conducted in 2009.  In this year, we were still advocating and requiring students to use rotating group roles and were assigning groups at the beginning of term, but did not shuffle the groups during the term.  We also asked students to report on locations that were convenient for them to meet and tried to create groups that were geographically overlapping with each other. 2.4.5.3 Workshops for Introducing Problem-Solving Skills Performing a full analysis of a novel complex problem requires many steps and correspondingly varied skills.  Students entering Physics 100 may not be   38 able to perform all of the separate tasks required to solve a complex problem. Indeed, the first years after the course transformation we found that the learning curve for tackling context-rich problems was too steep.  Students were immediately overwhelmed by problems that they did not immediately know how to solve, and the structured problem-solving method for approaching these problems was not helpful because it was entirely new to the students.  We realized that skills for performing the individual steps of the strategy needed to be learned as well.  For example, students that have only solved traditional end- of-chapter problems have never needed to determine which physics is necessary to solve a problem: it is always the physics contained in the chapter immediately preceding the problem.  Similarly, most students do not have practice in making assumptions in order to flesh out their physics model. To enable the students to develop the individual skills necessary to address novel complex problems we decided to construct the recitations as a “skills progression”, similar to that advocated by Teodorescu [20].  Van Heuvelen [21] also recommends that we should “provide students with explicit instruction in the individual skills used by experienced physicists when solving complex problems and then help them combine these skills to solve complex problems.” Based on these recommendations and the feedback on students’ initial difficulty with the full problem-solving method, we structured the first five recitation sessions as workshops focusing on one or two steps of the problem- solving method.  The goals for each workshop were to give students some practice with performing those steps of the method as well as to motivate why those steps were helpful for finding the right answers and developing understanding.  They were typically formatted as a series of exercises where students could perform each step in isolation, bracketed with full-class discussions to talk about how and why to perform these steps.  The course TAs were briefed on these workshops during the weekly TA meeting, and had a guide to help them structure these discussions.  The topic of each workshop is   39 summarized in Table 5 below, and the handouts and TA guides are reproduced in Appendix A. Recitation #  Skills Focus Problem- Solving Method Step Content Focus 1 Introduction to Structured Problem Solving Method. All n/a 2 The properties of Models and the role of Assumptions 3 & 4 Basic Kinematics 3 Identifying the Problem Goal and Relevant / Irrelevant Information 1 & 3 Conduction 4 Identifying Relevant Physics 2 Conduction and Radiation 5 Error-Checking and Sensemaking 6 Radiation and the Inverse Square Law Table 5:  Skill and content focus for the first five Physics 100 recitations.  These recitations were structured as workshops on developing students’ skills for individual steps of the overall problem-solving method. 2.4.5.4 Everyday Context Similar to other aspects of the course, the Physics 100 recitations were set in an everyday context as much as possible.  For example, problems about thermal conductivity were set in the context of household freezers or home insulation.  Problems about kinematics and energy were set in the context of everyday transportation: cars, cycling, and buses. As mentioned earlier, this was done to encourage students to see physics as relevant to the real world but there was an added pedagogical goal as well. Because we hoped to expose some of the more subtle ways that assumptions and modeling can help to simplify and address the complexities of real-world situations, we needed the students to be able to make assumptions about the problem contexts.  We assumed that setting the problems in an everyday world would enable the students to leverage their experience and intuition to develop models and to make sense of the physics.   40 Examination of this assumption is one of the key studies conducted in this thesis.  This study is described in chapter 6.  2.4.5.5 Motivation for Calculation Another important design feature of the Physics 100 recitations was that the circumstances of each problem were designed to motivate a true calculation, rather than a qualitative best guess.  Interviews with students had shown us that students feel that everyday problem solving rarely involves calculations. Rather, they said that they would tend to make decisions intuitively.  (see chapters 3 and 4 below for details of the interviews that informed this.) In order to demonstrate that quantitative problem-solving is a realistic and useful skill, we tried to construct situations where a calculation was truly called for.  In many cases that meant demonstrating ways in which physics calculations could save money or time.  We also tried to show how physics calculations could be consequential to peoples’ health, for example in planning a search for a missing hiker. In addition, an effort was made to make the result of the calculations consequential in that they would lead to a specific decision or action.  This is in sharp contrast to many physics problems which give no reason for performing a calculation other than the question itself. 2.4.5.6 Two Versions of Each Tutorial These everyday contexts were originally chosen by the instructional team. However, interviews after the first year of the course revealed that some students did not agree that the problem contexts were “everyday”.  (See chapter 4 below for a detailed discussion of these interviews).  For example, while all of the students were familiar with environments where the physics of home- heating is important, most of them have never paid a heating bill and were therefore not very interested in the details of household heat loss.  Several students reported that this was relevant to the real world, but not to themselves.   41 Another student objected to a problem that was about an astronaut whose suit was made of a thin garbage bag, saying that this problem sent a message that physics was only good for fantastical situations. This was an important comment, because initially several of our problems had been adapted directly from the University of Minnesota’s online archive for context-rich problems, many of which use an arbitrary or unrealistic motivation for calculation such as a “competition” or by stating that the student “wonders” what some quantity will be.  For example, another of our problems in the first year was adapted directly from this archive and opened with the phrase “You are a stunt coordinator for a movie shooting in Vancouver.” Based on students’ comments on these problems, I conjectured that students might react differently to a problem that was set in their everyday lives as they live them right now vs. a problem that was set in a hypothetical future job or situation.  In order to investigate this possibility each recitation problem in the 2009 year came in two versions that were mathematically identical but had slightly different cover stories.  The first version, which I called the Real Context, were problems set in the real world but which might not be strongly related to the students’ current lives or career goals.  The second version, which I called Personal Context, endeavored to place the problems within the students’ everyday lives and to make the calculations consequential.  The criteria for these two different types of problems are summarized on Table 6 below.   42 Personal Context Real Context Motivation Quantitative calculations are warranted.  i.e. the extra effort to get a numerical result is worthwhile Quantitative calculations may be asked in situations where commonsense estimation would be good enough Results of calculations are consequential. There will be some decision made or behavior will be different as a result of the calculation Calculations may be asked where no decision or action is implied by the result. Setting Setting is described in the first person. (i.e. “you”) Setting is described in the first person, but focus of the problem may be “a friend” Setting is drawn from everyday experiences that students are likely to have experienced Setting may be an unusual circumstance ( i.e. a contest, an experimental art work, a movie set) Setting may be a future job that is close to the career goals of the population (i.e. doctor, biologist) Setting may be a future job that is far from the career goals of the population (i.e. traffic consultant, engineer) The calculated result is a quantity that the students are able to interpret in terms of their own experience. (i.e. length, mass, fuel efficiency) Calculated result may be a physics quantity that students can only interpret in terms of other physics knowledge taught in the course (i.e. acceleration, intensity) Table 6:  Criteria for personal context and real context tutorials.  Each recitation problem in 2009 was developed in two versions with identical mathematical structure but slightly different cover stories. Students were split into two groups, each of which received one of these types of problems throughout the term.  Observations of students’ response to the problem cover stories and analysis of the CLASS scores of the two groups of students did not show any significant difference between these groups.  Because of the lack of result in initial investigations there is no further investigation of the impact of these contexts in the remainder of this dissertation. 2.4.5.7 Solutions as Worked Examples To offer students the best chance of learning from the recitations we offered students solutions that were structured according to recommendations from the Worked Examples literature [51,52].  Solutions contained mathematical, pictorial, and descriptive representations presented in an integrated fashion to help demonstrate the meaning and reasons for the various decisions made as a   43 part of the solution.  By providing these highly detailed solutions, we hoped to support students’ learning of expert-like problem-solving skills. 2.4.6 Changes to Lecture Many of the major changes to the Physics 100 lectures have already been noted: the topics covered in the course were changed and new lecture materials were developed to support this; the examples and questions used in lecture were written in an everyday context wherever possible; and a structured problem-solving method was used for in-class example problems.  The lecturers continued to use electronic response systems (clickers) to poll students with qualitative and quantitative questions during class.  There were also explicit lectures on the goals of this course transformation and the various methods that were being implemented in order to support it. The discussion on climate change was a significant deviation from the normal curriculum of an introductory course, but the style of presentation of this knowledge and the expectations for student learning followed a familiar pattern. Students were presented with diagrams representing physical phenomena and accompanying formulae, and these formulae were used to perform quantitative calculations. However, some of the discussion on climate change was conducted in a new way: the class moved away from straightforward calculations based on simple models with given information, and discussed some of the challenges in the application of mathematical climate models to the Earth’s climate. These discussions were largely qualitative, and covered several important issues such as the role of feedback in climate change, the history of Earth’s climate, and the evidence for and against humanity’s role in inducing climate change. Several common arguments about the nature of climate change and humans’ role in it were presented and critiqued by the instructor. In addition, several common statements about climate change that have been recently debunked in the literature were presented as myths of climate change. This material was not assessed on the course’s exams, but was intended to help students learn about the nature of scientific argumentation and critique.  This was   44 intended to help prepare students for the group research project, which required them to critique a scientific argument.  This research project is described in the next section. 2.4.7 Group Research Project An important change to the course was the incorporation of a final project where students work in small groups to quantitatively model and estimate an answer to a scientific question and present their results to their peers. Students choose project topics from a list provided by instructors and which were drawn from a magazine article suggesting various actions to reduce energy and greenhouse gas use, and students were asked to quantify the impact of these actions [53].  For example, projects have addressed the energy and greenhouse gas savings from using LEDs for city lighting and the energy benefits of paying bills online rather than by paper mail. Suggested topics are limited to those that fell within the scope of students’ ability to research and understand the topic and students are free to choose topics that were relevant or that interested them.  It was the intention of the course instructors that working on the final project would give students important practice in applying physics principles to real-world situations. There was little explicit instruction on the final project built into the course. The final project assignment was introduced and explained to student groups during the weekly recitation session in week 7 of the course. Instructors provided an example project and the project marking rubric, which emphasizes explicit discussion of physics but does not explicitly require students to use sophisticated physics models. Initially, the project was assessed via final presentations conducted in weeks 12 and 13 of the course and were graded by the course TAs and one of the course instructors. However, due to logistical pressures the presentation format was changed to a poster session in subsequent years.   45 3 STUDY OF CHANGES IN STUDENTS’ BELIEFS ABOUT PHYSICS One of the key goals of the transformation in Physics 100 was to improve the impact that the course had on students’ belief in the relevance of physics to the real world.  As mentioned in section 2.2.4, previous results of the CLASS survey [14] conducted in Physics 100 had shown that students’ belief in the relevance of physics to the real world declined over the course of instruction. In this study I use the results of the CLASS survey, which was conducted in 2006 prior to the transformation as well as in each year through 2010.  I also supplement the information gleaned from the CLASS survey results with the perspective of the course instructors. 3.1 Goal and Research Question The goal of this study is to examine the research question: Did the changes to Physics 100 improve students' beliefs regarding the relevance of physics to the real world? 3.2 Methodology The CLASS survey uses 42 statements about physics and learning with which the students are asked to agree or disagree on a 5-point Likert scale. Each statement has a clearly favourable and unfavorable response, as defined by the responses given by expert physicists and physics professors.  Student responses are scored by determining the percentage of items for which the student has given a favourable response, termed the % favourable score. For example, if a physics expert would respond ‘strongly disagree’ to a particular item, both ‘strongly disagree’ and ‘disagree’ are considered favourable responses. The response scale is collapsed in this way to avoid implicitly treating a ‘strongly disagree’ response as worth two ‘agree’ responses.  The % favourable score for a particular student can be calculated for both a pre- and post-test, which enables calculation of the % shift in a student’s favorable scores.  Similar calculations can determine the pre, post, and shift in each students’ % of unfavorable scores.   46 The items on the survey are grouped into categories by the test developers based on a factor analysis which identifies groups of response items that tend to correlate with each other.  For example, the items in the “Real-World Connections” category are listed in Table 7 below.  Each item on the survey can appear in more than one category, although some items are not strongly correlated with any others and are therefore not included in any category.  Such items are, however, included in the Overall scores. Question CLASS Questions:  Real-World Connections Category 28 Learning physics changes my ideas about how the world works. 30 Reasoning skills used to understand physics can be helpful to me in my everyday life. 37 To understand physics, I sometimes think about my personal experiences and relate them to the topic being analyzed. 35 The subject of physics has little relation to what I experience in the real world. Table 7:  CLASS response items in the Real-World Connections category.  These items are scored on a 5-point Likert scale:  strongly agree, agree, neutral, disagree, and strongly disagree.  Responses to items 28, 30, and 37 are considered favorable if the student chooses ‘strongly agree’ or ‘agree’.  Responses to item 35 are considered favorable if the student chooses ‘strongly disagree’ or ‘disagree’. The course instructors agreed that the four questions in the Real-World Connections category addressed one of the main goals in the course transformation:  that students should see physics as relevant to themselves and to their lives.  As such, improvement on this category was considered an important assessment of the success of the changes. The CLASS survey was administered as a pre- and post-test in each year of Physics 100 starting in 2006.  However in 2007 there was an important change in how students were recruited for the survey.  In 2006 the survey was strictly voluntary, resulting in predictably low numbers of participants, but in 2007 through 2009 the survey was mandatory, with students awarded 1% participation credit for completing both the pre- and post-tests.  This resulted in much higher participation rates as depicted in Table 8 below.  The effect of this change in survey recruitment on the selection of survey participants is explored in section 3.5.1 below.   47 Year Number of Students Number of matched pre/post responses Participation rate 2006 717 91 13% 2007 711 384 54% 2008 708 481 68% 2009 811 506 62% Table 8:  Class sizes and participation rates for the CLASS survey in Physics 100 from 2006 to 2009.  In 2006 participation in the survey was strictly voluntary.  In 2007 through 2009 1% of course grade was given for participation in both the pre- and post-tests. In all years, the survey was administered online via the course’s website. 3.3 CLASS Survey Results The CLASS survey results for 2006 − 2010 are shown below.  Because of variations in the incoming classes each year, I focus primarily on the shift scores, which identify how students’ responses changed over the course of the term.  Each bar reports the average for the whole class, and the error bars show the standard error for each score.  The main categories of interest were the Real-World Connections category and the Overall score, which are summarized on Figure 1 and 2 below.    48   Figure 1:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey.  The shifts are calculated as %shift = %favorable_post - %favorable_pre for matched pairs of pre/post surveys.  Figure 2:  Pre-post shift in unfavorable responses for the Overall and Real-World Connections categories on the CLASS survey.  The shifts are calculated as %shift = %unfavorable_post - %unfavorable_pre for matched pairs of pre/post surveys. -8% -6% -4% -2% 0% 2% 4% 6% 2006	
  Shift	
   2007	
  Shift	
   2008	
  Shift	
   2009	
  Shift	
   2010	
  Shift	
   Favorable Overall Favorable Real-World Connections -4% -2% 0% 2% 4% 6% 2006 Shift 2007 Shift 2008 Shift 2009 Shift 2010 Shift Unfavorable Overall Unfavorable Real-World Connections   49 An examination of all five years of data using ANOVA shows that there are some significant differences between the years.  Pairwise Tukey Honest Squared Differences (HSD) comparison shows that the Favorable Overall score for 2009 is significantly lower than it was in 2008 (p<0.05), and the score in 2010 is significantly greater than it is for 2009 (p<0.01).  Similarly, the Unfavorable Overall shifts for 2009 were greater than those in 2008 (p<0.05) and in 2010 (p<0.001).  There were no significant differences between any other pair of years. ANOVA analysis of the Real-World Connections scores showed no significant effect of the course year on either of the Favorable or Unfavorable scores (p>0.05). Because the course transformation was explicitly intended to improve students’ perceptions of the relevance of physics to the real world, we were surprised to see no significant difference between the 2006 (pre-transformation) results and the 2007 (post- transformation) results.  After the appearance of some improvement in the 2008 Real- World Connection results, we were also surprised to see the results for 2009 decrease and become comparable to the initial pre-transformation scores. 3.4 Analysis of Pre-Course Scores One of the factors that I examined for connection to these surprising results was the variation in the incoming students’ survey scores.  The pre-course scores for 2006 – 2010 are shown in Figures 3 and 4 below.   50  Figure 3: Pre-course Favorable CLASS survey scores for 2006 - 2010 in the Overall and Real-World Connections categories  Figure 4:  Pre-course Unfavorable CLASS survey scores for 2006 - 2010 in the Overall and Real-World Connections categories An ANOVA analysis revealed some significant differences in the pre-course scores between different years.  For the Overall Favorable category the pre- scores in the 2008 year are significantly lower than in the 2007 year (p<0.05).  For the Overall Unfavorable 45% 50% 55% 60% 65% 2006 Pre 2007 Pre 2008 Pre 2009 Pre 2010 Pre Overall	
   Favorable Real-­‐World	
  Connections	
   Favorable 10% 15% 20% 25% 30% 2006 Pre 2007 Pre 2008 Pre 2009 Pre 2010 Pre Overall Unfavorable Real-World Connections Unfavorable   51 category the 2006 pre-scores are significantly higher than the 2007, 2009, and 2010 years (p<0.05).   For the Real-World Connections Favorable category the 2007 and 2010 years are significantly higher than the 2006 and 2008 years (p<0.05).  Finally, for the Real- World Connections Unfavorable Category, the 2006 and 2008 pre-scores were both higher than the 2010 score To investigate the influence of these variations in pre-score on the % shifts I examined the correlation between these variables.  For the Overall category the correlation between shift and pre-scores is R=-0.40 and for the Real-World Connections scores it is R=-0.48, both of which are significant at the p<0.001 level.    These correlations may be partly due to the floor effect:  students who begin the course with low beliefs haven’t as much to lose on the post-test, and are therefore not capable of posting very large negative shift scores. Alternatively, one might imagine that students with low pre-scores are less likely to engage with the course in such a way that their personal beliefs about physics and learning might be challenged and thereby evolve.  In either case, it is clear that students with higher pre- scores are more likely to have strong negative shifts. To take the effect of these variations in pre-scores into account, I conducted a multivariable regression analysis to examine the CLASS shifts with the pre-scores and the course year as predictors.  However, when the pre-scores were included in the model the course year was not determined to be a significant predictor of shifts, suggesting that the variation observed in section 3.3 above was largely due to changes in the pre-scores rather than in changes due to the course curriculum. In the following sections I examine several other factors to investigate the surprising lack of improvement in the CLASS scores in the 2007 year and the surprising decrease in scores in the 2009 year. 3.5 Analysis of Lack of Change in 2007 In order to examine whether some other difference in the course could be contributing to these results, I examined several factors that could impact the CLASS scores.   52 3.5.1 Differences in Population due to Recruitment Method Because of the difference in recruitment method and participation rates there was some concern that the student population participating in each year might represent different segments of the class.  In particular, in 2006 the survey was voluntary and in 2007 it was made mandatory. In order to assess the impact of the differing requirements for survey participation, I compared the course grades of the participating students to the overall average course grade for both 2006 and 2007.  The surveyed sample’s average grade was 5% higher than the average grade of the entire class in 2006 and was 4% higher than the entire average grade of the entire class in 2007.  Because the difference in average grades were so similar, I could not conclude from the course grades that a difference in sampled population could be responsible for the lack of significant change in the CLASS shift from 2006 to 2007. 3.5.2 Faculty Changes Another important factor to consider in interpreting these results is the impact of the faculty changes from one year to the next.  The course is taught in three different sections, and in the period studied a total of six different faculty taught in the course. As well, there were different participation rates in each of the course sections which weighted the results of that section more heavily and may have influenced the overall results.  The number of participants in each section and the faculty teaching each section are illustrated in Table 9 below.  Number of CLASS Participants by Section  Legend Year Section 101 Section 102 Section 103 Total N (whole class) Instructor A 2006 27 36 28 91 717 Instructor B 2007 97 152 135 384 711 Instructor C 2008 145 138 198 481 809 Instructor D 2009 168 157 181 506 811 Instructor E 2010 137 102 116 355 734 Instructor F Table 9:  Instructor changes and number of CLASS survey participants in Physics 100 from 2006 to 2010.   53 To examine whether instructor effects might be responsible for the lack of improvement in the CLASS scores after the course transformation, I compared the 2006 and 2007 results using only the data from the instructors that taught both years. The results again show no significant difference between the 2006 and 2007 results. (See Figures 5 and 6 below)  Figure 5:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey for the two sections that had the same instructor for 2006 and 2007.  Figure 6:  Pre-post shift in unfavorable responses for the Overall and Real-World Connections categories on the CLASS survey for the two sections that had the same instructor for 2006 and 2007. -10% -8% -6% -4% -2% 0% 2% Favorable: Overall Favorable:  Real World Connection 2006 (N=55) 2007 (N=232) -2% 0% 2% 4% 6% 8% 10% Unfavorable:  Overall Unfavorable:  Real World Connection 2006 (N=55) 2007 (N=232)   54 3.6 Analysis of Decline in 2009 Despite the appearance of some significant improvement in the 2008 Real-World Connection results, the results for 2009 and 2010 were comparable to the initial CLASS scores prior to the course transformation in 2006. The change in the CLASS shifts was surprising and disappointing.  The results were surprising because the course instructors had not endeavored to make any major changes in the 2009 year.  While there were some refinements made to each of the lecture, lab, and recitation materials, the course instructors felt that on the whole the 2009 course was taught in the same way as the 2008 course.  The CLASS shifts in 2009 were disappointing because the 2008 results were statistically significantly better than the 2007 results, and if the improvement demonstrated in 2008 had continued it would have been good evidence that the refinements to the course in 2007 had paid off and overall the course transformation had been successful. To try to understand why the shifts in 2009 were not as favorable as in 2008 I investigated other possible factors that might explain the decline.  I examined several possible explanations for these changes: the change of one course instructor from 2008- 2009; changes in the demographic composition of the students in the course; and differences in timing of the post-course survey. 3.6.1 Instructor Changes One possible cause was the change of instructor in one section of the course (see Table 9 above).  To investigate the possibility that this one section was responsible for the major shift seen from 2008-2009 we looked at the CLASS results for only the two sections that had the same instructor for all three years.  The results, summarized below in Figure 7 and 8 show that the 2009 results for both of these instructors were statistically indistinguishable or, in the case of the Overall favorable %shift, worse than their 2007 results.  Because the pattern of declining results in 2009 occurred even in the two sections that had the same instructor, I rejected the hypothesis that the decline was caused by the new instructor.    55  Figure 7:  Pre-post shift in favorable responses for the Overall and Real-World Connections categories on the CLASS survey for the sections that had the same instructor each year.  Figure 8:  Pre-post shift in unfavorable responses for the CLASS survey Overall and Real- World Connections categories for the sections that had the same instructor each year. -8% -6% -4% -2% 0% 2% 4% Favorable:  Overall Favorable:  Real World Connection 2007 shift 2008 shift 2009 Shift 2010 Shift   56 3.6.2 Student Demographics The observed variation in CLASS pre-scores suggested that there might have been an important difference in the student population that year that could explain the decreased CLASS shifts.  I investigated the hypothesis that perhaps there had been a significant change in the demographics of the students taking the course in the 2008 year that could help to explain the difference. While there are correlations between students’ program and their Overall CLASS shifts, a linear model of the CLASS scores that takes students’ program, section, and the course year into account does not find that these demographic factors are significant predictors of students’ CLASS scores.  Therefore, I could not conclude that the demographic differences in the student population in 2009 were responsible for the CLASS shifts. 3.6.3 Timing differences Because we were concerned that the scores on attitude and conceptual surveys could be influenced by their timing in the term we intended to always give the survey during the first week of class and the last week before the final exam.  However in 2009 due to logistical errors it was necessary to leave the post-survey open for two full weeks.  In order to investigate the possibility of the extended post-testing period affecting the shift results I compared the CLASS scores for the participants that completed the post-test in the first week of the testing period against the scores for the participants that completed it in the second week of the testing period.  These comparisons are depicted in figures 7 and 8 below.   57  Figure 9:  Favorable CLASS shift results for the 2009 year, broken down by the week in which students completed their post-test.  In this year, the post-testing period was open for two weeks, instead of the usual one week.  Figure 10: Unfavorable CLASS shift results for the 2009 year, broken down by the week in which students completed their post-test.  In this year, the post-testing period was open for two weeks, instead of the usual one week.  Because there was no statistical difference in the category scores of the two groups we concluded that this effect could not be responsible for the different CLASS shifts between 2008 and 2009.  Therefore the differences in timing of the post-test were discarded as a possible explanation for the decreased CLASS shifts in 2009. -10% -8% -6% -4% -2% 0% 2% Favorable:  Overall Favorable:  Real World Connection Week 1 Shift Week 2 Shift 2009 Shift -4% -3% -2% -1% 0% 1% 2% 3% 4% 5% 6% Unfavorable:  Overall Unfavorable:  Real World Connection Week 1 Shift Week 2 Shift 2009 Shift   58 3.7 Summary According to my analysis, the CLASS shift in the Overall and Real-World Connections category did not significantly improve over the years 2006 − 2010 and on average remained negative in both categories, indicating that students’ beliefs and attitudes towards physics were less favorable after taking the course.  Investigations to examine the effect of changes in faculty, different recruiting methods for the CLASS survey, changes in student demographics, and changes in the timing of the post-test did not find that any of these factors explained this result.  The results of this survey suggest that presenting physics in an everyday context is not sufficient to promote students’ perception of the relevance of physics to the real world.   59 4 EXPLORATORY POST-COURSE INTERVIEWS In order to investigate students’ overall perceptions of the course changes and to explore their impact on student’s attitudes towards the real-world relevance of physics nine students were interviewed after the completion of the first year of the transformed Physics 100 course in 2007.  This study used CLASS statements as interview prompts to examine students’ perceptions of the relevance of physics to the real world in detail. Based on their responses, I identify several ways in which students perceive the relevance of physics to the real world and factors that promote or inhibit the perception of relevance.  One key finding was that students judge relevance according to their immediate circumstances and career goals, and therefore problems that relate to implausible or hypothetical future situations may not be seen as relevant. 4.1 Goal and Research Questions These interviews had several goals.  Firstly, because we were using the CLASS survey as a measure of the course’s impact on students’ perception of the real-world relevance of physics, I wanted to probe the details of how students interpreted the wording of the CLASS questions.  Even though students’ interpretation of these questions had been validated by the survey’s original authors [54] I wanted to re-validate using our local population.  Therefore I address the research question: 1. How do students interpret the CLASS survey questions? Secondly, I wanted to inquire about students’ overall impressions of how the Physics 100 course affected their perception of the relevance of physics to the real world. Through general conversation about the course, I hoped to uncover the ways that students see physics as being connected to the real world, and explore the factors they cite as affecting their beliefs.  This goal motivates the next two research questions: 2. In what ways do students see physics as related to the real world? 3. What factors do students cite as affecting their belief in the relevance of physics to the real world?   60 Finally, I wanted to obtain general feedback on the course elements that would help us to develop subsequent versions of the course.  Although this was an important goal of these interviews, it is not relevant to the research questions addressed in this thesis, so I will not focus on the information gleaned in this regard. As described in Table 1, the research questions mentioned above offer a perspective on the overall dissertation research questions. 4.2 Selection of Participants Because I wanted to gather a representative sample of students’ impressions of the course I attempted to recruit an interview cohort that would span the variety of student responses towards the course on the CLASS instrument.  Therefore,  participants who had extremely high or extremely low shifts in scores on the CLASS Overall and Real-World Connections categories during the 2007 year were invited to participate.  Participants were contacted by email, and all students that volunteered were invited into the study. Because the intent of the interviews was merely to probe the variety of student attitudes and suggest possible reasons for shifts on the CLASS scores, it was deemed unnecessary to have a demographically representative sample.  All nine of the participants were female.  Six of the nine participants were life sciences majors, which approximately matches the course demographics for that year. 4.3 Interview Protocol The protocol used was a semi-structured conversation using CLASS statements as prompts.  I began each interview with a few demographic and background questions, and tried to establish a nonjudgmental rapport with the participant.  Then I proceeded to read out individual CLASS questions and ask the students to indicate their agreement on the standard Likert scale of Strongly Agree, Agree, Neutral, Disagree, or Strongly Disagree. Sometimes students would make a choice on this scale, but often they would spontaneously comment on the question or the subtleties of their opinion.  Before proceeding to the next question I always asked students to provide their response to the current item.   61 In this manner, each student was interviewed on all four of the CLASS Real-World Connections questions and some of the Problem-Solving questions.  Due to time restrictions, not every question was asked of every student.  The questions in this protocol are summarized in Table 10 below.  As a novice interviewer I was concerned that using free-form follow-up questions would unduly bias the participants’ responses, so I used a rigid set of follow-up questions to probe students’ reasons for their CLASS item responses. These follow-up questions are listed in the complete interview protocol in Appendix B.    62  Demographics Questions CLASS Question How old are you? What is your major in school? Can you tell me a little about why you chose that major? What year are you in? What is your favorite class?  Why? What do you want to do when you graduate? Why did you volunteer for this study? CLASS Questions:  Real-World Connections Category “I’ll read these statements to you, and I’d like you to reply by letting me know whether you agree or disagree with them on a scale of 1 to 5, 1 being strongly agree, and 5 being strongly disagree. Learning physics changes my ideas about how the world works. 28 Reasoning skills used to understand physics can be helpful to me in my everyday life. 30 To understand physics, I sometimes think about my personal experiences and relate them to the topic being analyzed. 37 The subject of physics has little relation to what I experience in the real world. 35* CLASS Questions:  Problem-Solving Category “I’ll read these statements to you, and I’d like you to reply by letting me know whether you agree or disagree with them on a scale of 1 to 5, 1 being strongly agree, and 5 being strongly disagree.” I do not expect physics equations to help my understanding of the ideas; they are just for doing calculations. 13* If I get stuck on a physics problem on my first try, I usually try to figure out a different way that works. 15 Nearly everyone is capable of understanding physics if they work at it. 16 I can usually figure out a way to solve physics problems. 34 When studying physics, I relate the important information to what I already know rather than just memorizing it the way it is presented. 42 After I study a topic in physics and feel that I understand it, I have difficulty solving problems on the same topic 5* If I don’t remember a particular equation need to solve a problem on an exam, there’s nothing much I can do (legally!) to come up with it. 21* If I want to apply a method used for solving one physics problem to another problem, the problems must involve very similar situations. 22*  Table 10:  Summary of questions in exploratory post-course interviews.  Students were asked to read these questions aloud, comment on whether they agreed or disagreed with them on a 5-point Likert scale, and asked to explain their reasons for their responses. Responses to questions with a * next to the item number are considered favorable if the student disagrees with the statements.   63 4.4 A Methodological Comment In order to check the validity of these interviews, the students’ verbal responses to the CLASS questions were compared to their actual responses on the CLASS survey.  A good match between the students’ CLASS post survey responses and their responses in the interview would support the idea that: 1.  The students’ responses on the CLASS survey were stable from the time they took the post survey until the time they were interviewed (approximately 3 months) and 2.  The students’ responses in the interview context draw upon the same attitudes and opinions as their responses in the survey. A comparison of the CLASS post scores and responses given in the interview reveals that these match only 57% of the time.  For the purposes of this comparison, a response of “Strongly Agree” is considered a match with the response “Agree” and similarly “Strongly Disagree” and “Disagree” are considered a match.  This agreement rate is better than chance, but still indicates that either or both of condition 1 and condition 2 above are not met. Either of these conditions might plausibly be violated.  As a novice interviewer, I may have failed to create the framing of “mutual enquiry” recommended for this type of clinical interview [55] and the students may have been framing these interviews as evaluative tests.  In addition, it is reasonable to propose that during the delay between the CLASS post-test and the interviews that these students’ attitudes towards physics legitimately changed.  Many of them were enrolled in another physics course, which could certainly affect their perspectives.  We must consider the possibility these CLASS questions do not measure a students’ attitude as a unitary and stable property.  Students are humans, and as such are sensitive to priming effects and liable to answer the same question differently on different days. This suggests that we should not draw conclusions from these interviews about the reasons for the particular responses of these particular students on their CLASS post-test. However we can take these students as representatives of their classmates, and treat their responses as representative of viewpoints that may arise in the population at large.   64 4.5 Interview Results  The principal results of these interviews are an illustration of the diverse ways in which students conceive of the notion of “real-world connections” and of the notion of “reasoning skills used in physics.”  As well, students gave us some specific feedback on their impressions of what constitutes a real world context that informed future development of the course and the recitations. 4.5.1 Diverse Interpretation of “Reasoning Skills Used to Understand Physics” Students expressed a wide diversity of interpretations of the phrase “reasoning skills used to understand physics”, used in CLASS question 30.  This variety of interpretations reveals students’ different expectations about the nature of physics reasoning skills, as well as their implicit epistemological stance on the potential connections between physics and the real world. In the following sections I will give examples of several different ways that students interpreted the concept of “reasoning skills used to understand physics” and illustrate each example with an excerpt from the interviews. 4.5.1.1 Reasoning Skills as Problem Solving Several students’ answers reflected their interpretation that “reasoning skills used to understand physics” are a form of critical thinking, problem-solving skills, or the specific problem-solving strategy used in the Physics 100 recitations. One student mentioned applying mathematics to word problems as well as critical thinking, which he describes as being distinct from specific physics content knowledge. Alberto: With physics problems you're generally taking words and applying some kind of math to them. Interviewer: Do you find those skills useful to you... Alberto: I'm just thinking if they're useful...I think they help like problem-solving skills in general, they are good skills to practice.   65 Interviewer: Why? Alberto: Well in order to be a successful person you need to be able to critically think and I think that's the main thing they try to teach at the university. You're not going to remember facts, and you're not going to remember physics formulas.  The second student cites reasoning skills as being related to looking logically at questions posed in class, and making good use of formulae. Interviewer: (prompting for agreement / disagreement) “Reasoning skills used to understand physics can be helpful to me in my everyday life." Joyce: The first thing that comes up is like problem-solving skills and just like when you look at a question, the logic of it, and that's useful in other classes. I'm not sure if you'd consider that everyday life. It's certainly my everyday life. […]  Like math or even other courses like biology when you're given a statistical analysis of something. Population density or something like that. So yes. problem-solving skills. Interviewer: […] Do you think that you developed those reasoning skills over the course of Physics 100 or do you think you mostly have the same before and after taking the course? Joyce: Well for physics it's applying the question logically to like a formula. At least for that course it kind of is. And so a lot of courses you're applying it to a formula. I'm learning physical chemistry right now and that's pretty much the same kind of idea.  An interesting aspect of the above response is that she interprets “everyday life” to mean “other courses at school”.  This seems quite a sensible interpretation for a student that, like most other undergraduates, spends most of her time engaged in school-related activities.  It also seems that she is using Physics class as a standpoint from which to refer to her other courses. The third student interprets reasoning skills as a process or strategy used to solve problems. Interviewer: Reasoning skills used to understand physics can be helpful to me in my everyday life.   66 Leslie: Yes. Interviewer: Why do you say that? Leslie:  Well I guess there are two aspects of that. There’s the actual mental or psychological, what ever you want to use, process of solving problems. So it kind of develops your ability to not just narrowly look at one idea, but to see all the aspects bring together what the, know which aspect you need to take into account. Some you need to disregard and some you have to, and then follow kind of a logical process to get your results. [Continued below] 4.5.1.2 Reasoning Skills as Physics Content Leslie, the student quoted above, also mentioned the relevance of physics content knowledge to other sciences. Leslie: [Continued from above] Then there's also the physics side. What we're being taught. It's already very clear that it's applicable in our normal life, just that we need, each field needs to recognize that it's applicable […] if you have biology students in there and they aren't connecting to what's being taught. They might not see that connection and, I dunno, I guess I'm a huge interdisciplinary person. And same with chemistry and all that. I don't see how physics can be left out of any science I guess is what I'm saying. It's one of those fundamental skills that needs to address each portion. If it’s being taught well rounded with all the aspects, then you should be able to use it in anything, If a teacher in a biology class asks a question about bird flight, you should draw from your physics course and say how is this going to work.  In this quote, Leslie specifically highlights that she believes that physics is a “fundamental skill” that offers knowledge that is useful in other scientific disciplines.  In this case, she is implicitly treating the prompt of reasoning skills used to understand physics as being physics knowledge that can be applied elsewhere. This interpretation of “reasoning skills used to understand physics” in terms of physics content knowledge was shared by Florence, who mentioned this type of knowledge primarily in her explanation of why she doesn’t think about physics in her everyday life Interviewer: “Reasoning skills used to understand physics can be helpful to me in my everyday life.”   67 […]  What type of reasoning skills do you think of when you hear that statement? Student: Well physics reasoning, why things work the way they do. Why, how come a ball that's sitting, a ball falls and it doesn't keep on bouncing so that kind of thing.  Her emphasis on thinking about the dynamics of a bouncing ball show that she is framing the question about reasoning skills in terms of the utility of physics content knowledge in everyday life. 4.5.1.3 Reasoning Skills as Common Sense The above responses dealt with skills and knowledge that were expected by the instructors of this course.  However several students interpreted this phrase in an unexpected way.  The next two students’ answers indicate that they felt that “reasoning skills” were more closely connected to everyday reasoning or common sense than to physics. First Student: Interviewer:  What kind of reasoning skills do you think of when you hear this question? Nadine:  My mind is at a complete blank right now because it seems like we didn't get any reasoning skills from the course. […] Interviewer: So what would be a reasoning skill? […] Nadine: Reasoning skills that would be involved in your everyday life I guess. Physics 100 just didn't seem like it really affected my life and my choices.  Second Student: Interviewer: What do you think of when you think of reasoning skills? What did you think of right away? Patricia: I don't know, somebody not stupid. […] Yeah there's a certain amount   68 of reasoning like cars can't go like a gajillion miles per hour. <Sarcasm> Oh, let’s THINK about that. </Sarcasm> I dunno, yeah there's reasoning skills.  You should be logical but you should be logical in *everything*. Interviewer: It seems like you're saying that you walk into the course and you have a certain perspective on reasoning skills. And so my question is. Do you think that you learned those skills? Patricia: Yeah probably. But I think I learned them through trial and error in real life and certain people's personalities have more reasoning skills and people are inherently like, “Be logical, you can't be a dreamer”. Whatever. So that's applicable to life. If you see things in a way that's like “This can happen I see this happening, this is reasonable,” I'm sure you'll be reasonable in most aspects including education and physics.  For these students, it seems that the phrase “reasoning skills used to understand physics” was not distinct from their general perception of reasoning skills used in everyday life.  This interpretation would make the CLASS item 30 “reasoning skills used to understand physics can be helpful to me in my everyday life” almost tautologically true.  For these students, a positive response on that item says more about their perception of the term “reasoning skills” than it does about the connection between physics and everyday life. 4.5.1.4 Reasoning Skills as Exam-Taking Strategies One student interpreted reasoning skills as primarily exam-taking skills. Samantha:  The organizational skills that I learned in physics can be helpful to me. But that has very little to do with the physics. Interviewer: What other types of reasoning skills do you think, come to mind. Samantha: There's a lot of process of elimination. Because I didn't necessarily understand it but I was like: it can't be that, that doesn't make sense here. There’s a lot of process of elimination which is a good skill to be able to do. There's a lot of just like, thinking back to trying to write the exam, trying to the interpretation of questions. Trying to understand what he's actually asking.  For Samantha, “reasoning skills used to understand physics” were primarily utilitarian skills such as interpreting exam questions and applying the process of   69 elimination.  This suggests that for her understanding physics is the same as doing well on the physics exam. This interpretation is supported by earlier statements that Samantha had made about her approach to the course’s open-book exam.  She described how her exam preparation consisted almost exclusively of indexing and organizing her notes so she could look things up quickly during the exam.  For her, “understanding physics” was very closely connected to “understanding how to pass the physics exam.” 4.5.2 Conceptions of Physics in the Real World Students’ responses in these interviews illustrated a wide diversity of ways in which students perceived connections between classroom physics and the real world. This was especially important because the course instructors had been thinking of “real-world connections” in an implicitly unitary fashion and had assumed that by addressing important questions presented in familiar contexts the students would see the importance of physics in their everyday lives.  These interviews made it clear that these assumptions needed to be critically re-examined. There emerged from the interview transcripts several different varieties of the notion of a connection between physics and the real world.  Interviewees evidenced that they could believe strongly in one form of real-world connection while rejecting another.  In the following sections some of the different ways that students described connections between physics and the real world are described and illustrated with excerpts from the interviews. 4.5.2.1 Physics as Relevant Environmental Issues When prompted by the CLASS statements many students expressed their belief that physics is connected to the real world.  However, they did so in a way that connects it to large environmental issues rather than everyday life. Because climate change was a key aspect of the course curriculum, it is not surprising that students would mention the connection between physics and this issue.   70 The quote below illustrates how a student may think of real-world connections in physics at the level of a large issue. Beryl: [Learning about physics] opens my eyes to global warming and the other crisis in the world. So I'm more aware of them if I learn more about them and like if you don't know about then, if you don't really know that something like this exists.  The next quote illustrates how one student distinguishes between the relevance of the overall topic as opposed to the relevance of the component elements of the physical theory of that topic that was presented in class. Debby: I think it connects, just not to a point where you would actually think of it in day to day life. Like global warming, pretty big issue, it matters. But then when we're talking about the solar flux or something, I don't remember, it just didn't seem relevant.  This type of split illustrates the fine nuances of students’ perceptions of “real-world connections”; their perception of the relevance of physics to the real world may be very fine-grained, and susceptible to highly personal preferences. This distinction in students’ and experts’ epistemologies of physics could be explained in terms of the integration of their knowledge.  An expert who sees an overall theory as relevant and who knows that the component elements are necessary for understanding and utilizing that theory is likely to see the component elements as relevant as well.  The integrated, hierarchical knowledge structure of the expert allows them to attribute the same meaning and relevance to the pieces as to the whole.  However, for a student who perceives physics more as “disconnected facts”, the relevance of the larger theory may not imply relevance of the component elements.   71 4.5.2.2 Physics as Relevant vs. Physics as Worth Noticing When explicitly prompted to discuss the relevance of “the subject of physics” to everyday life, most of the interviewed students reported that they believe it is deeply relevant to the action of the real world.  There is a sense, perhaps merely derived from a perception of science as an authority on the nature of the world that physics is somehow at the root of many everyday phenomena.  We can see this in the first part of the quote below. Florence:  The subject of physics itself I'm sure is absolutely relative to my life. I'm sure that all the things that I do in life have some physics component behind them…  However, the second part of that same quote reveals another conception of how physics might connect to the real world. Florence: …but at the same I don't think about it like that. I feel like it doesn't but in my head I know that somehow it does. I'm obviously walking, there's a force of gravity on me so that's physicsy but like but that's not what I think about when I walk across the room. I just go through my days, and I don't really give it a thought. Or like when I turn on a lamp, I don't think how electricity is traveling.  Another student’s quote demonstrates this same split between belief in the relevance of physics and the perception of that relevance. Vivian:  I know [physics] is relevant and everything, [the instructors] make everything applicable to the real world which is important. I am not passionate about it so I don't really make it relevant. I could make it relevant if I was really interested in it. 4.5.2.3 Physics as Calculations In response to being asked to agree or disagree with one of the CLASS prompts,  another interviewee reported that she believes physics has little relation to the real world based on her conception of the nature of physics as mostly calculations.   72 Interviewer:  (prompting student to agree or disagree with the following statement)  “The subject of physics has little relation to what I experience in the real world.” Helene:  I would say I agree with that. Because I don't know, what I think of physics is mainly like calculations and things like that. And it doesn't seem like I ever do them in real life.  Rather than saying she simply doesn’t think about it, she is expressing her belief that the main activities in real life are distinct from the main activities in physics.  For her, it seems to make little sense to connect physics and real life. 4.5.3 Factors Influencing Perception of Relevance Two main factors that influence students’ perception of the relevance of physics to the real world emerged from these interviews.  The first is that students in this population tended to judge relevance based on their immediate lives or stated career goal.  The second factor is that students reported that problems with a fantastic or humorous setting sent a message that physics was only applicable in such settings. These factors are discussed in the sections below. 4.5.3.1 Personal Perspective on Real-World Relevance These interviews also revealed some general differences in the students’ and instructors’ perceptions of what constituted a real-world connection in physics.  As mentioned above, students’ comments indicated that they made a clear distinction between relevance to the world at large and relevance to themselves personally.  This result was not surprising as it echoes the split found by Adams et. al in their development of the CLASS survey [54].   However an important pattern that emerged was that students generally assess relevance to themselves based on direct past experiences, current interests, and current career goals.  They reported little interest or feeling of relevance for physics set in real-world contexts that do not fall into these categories.  Several students commented that they did not see strong connections in problems or contexts that were not relevant to them at the current time.   73 A good example of this is the case of driving.  Several students complained that using driving as a context for mechanics problems isn’t directly relevant to students that don’t have a drivers’ license or a car. The student below describes how she doesn’t think about acceleration or velocity because she doesn’t drive. Wendy: We don't really think about [kinematics] I guess. I don't drive yet. So then can't really apply the kinematics of like acceleration or velocity or anything. You don't really think about it when you're just walking or taking buses or something like that. This result was in conflict with the instructors’ expectations, and started to reveal the difference between the students’ and instructors’ perception of real- world relevance.  This underscored the importance of understanding and addressing the students’ beliefs rather than the instructors’ beliefs about these contexts.  Subsequent polling of the students revealed that the majority of Physics 100 students didn’t drive a car on a regular basis, which prompted a shift within the second year of the course towards more kinematics questions related to busing, cycling, and walking. These results also prompted an increased focus on contextualizing the course physics in terms of the students’ everyday life at the present time, rather than their hypothetical future. 4.5.3.2 Interpretation of Realism of Tutorials Another result that is important to anyone writing physics word problems was that students found some of our recitation problem situations unrealistic or fantastic.  Two problems in particular were cited in the interviews: the first involved calculation of thermal energy balance for an unfortunate astronaut who had only a black garbage bag for a space suit, and the second involved calculation of friction for a stuntman stuck on the front of an accelerating train. For the complete text of the recitation problems see appendix A. One interviewee explained that she felt that problems such as these send the message that physics is only relevant in unrealistic or comical situations such as these.   74 Valerie:  I remember one question on whether heat would be absorbed or reflected, the astronauts used garbage bags instead of their suits. But that would be unrealistic. […] I think these kinds of questions, they seem so unrealistic, it makes us think that physics doesn't relate to us. And it can't relate to us, because the recitation problems are connecting us to unrealistic situation. They are kind of saying it can't connect to everyday situation. But instead they have to make it into an unrealistic situation. Kind of comical, astronauts in garbage bags, a comical situation instead. Considering the widespread use of fantastical or humorous situations as contexts for physics problems, this comment has important instructional implications, which are discussed in the Chapter 7 of this dissertation. 4.5.4 Summary In this chapter I used interviews to examine students’ responses to the CLASS survey questions that contribute to the Real-World Connections and Problem-Solving categories.  I demonstrated several different ways that students interpret the phrase ”reasoning skills used to understand physics” which is used in question 30 of the CLASS survey. I also identified several different interpretations of the ways in which physics is relevant to the real world.  In general, students tend to distinguish between relevance to others and relevance to themselves and judge relevance to themselves based on their immediate current circumstances and career plans.  This shows the importance of learning about students’ everyday lives and career plans in order to demonstrate relevance to them.   75 5 STRUCTURED REAL-WORLD RELEVANCE INTERVIEWS The unstructured post-course interviews had shown the variety of ways that students might perceive relevance to the real world, as well as giving us a look at the kinds of problem features that novices react to in developing those opinions.  While students’ professed opinions and their actions with respect to a particular problem in the real world may not be the same [11,56,57], their stated opinions still offer valuable insight into their beliefs. Because of the unstructured nature of the study described in chapter 4 it was rare for more than one student to comment on the same physics problem or problem feature.  In order to learn more about which problem features promote the perception of relevance for a wide variety of students, a more systematic study was required.  In this study, a diverse cohort of students were interviewed on their opinions on a set of scientific problems that were designed to span many different features that could prompt a judgment of relevant or irrelevant to the real world. Students were asked several questions about each problem and I performed a qualitative analysis of their responses to identify the features that each student cited as motivating their perception of relevance.  To explore which problem properties were statistically correlated with higher overall ratings of a problem’s connection to the real world, I coded the interview results and performed an ANOVA analysis. Below, I describe the goal, methodology, and results of this study. 5.1  Goal and Research Questions As stated above, the purpose of these interviews was to systematically examine students’ opinions of the real-world relevance of a fixed set of problems.  The features that students cited in their justification for their judgments help to answer the following research question: What features of scientific problems do students see as connected to the real world?   76 5.2 Methodology 5.2.1 Interview Cohort These interviews were conducted in the spring of 2009.  To ensure a diverse cohort the interview participants were recruited from the general student population via poster advertisements that encouraged students who “Love Science” or “Hate Science” to apply.  Posters were placed in the student union building, the main university bookstore, and in the university libraries.  In addition, participants were recruited via in-class recruitment from introductory physics and astronomy classes that served non-major populations.  The final cohort of participants was ten undergraduate students who spanned a wide variety of declared majors and experience in university.  See Table 11 for a summary of these students’ backgrounds. As a part of the interview protocol (see below), the participants were asked to respond to the four questions in the CLASS survey’s real-world connections category. The last column of the table gives the students’ scores in this category.  A higher number means the student reports more expert-like beliefs about the relationship between physics and the real world.  Although I attempted to recruit a cohort that was equally distributed in terms of their stated beliefs about the real-world relevance of physics, only two of the students scored less than 50% on the CLASS questions administered in the interview.   77 Gender Yr. Major % Favorable on CLASS Real- World Connections scale F 3 Clinical Psychology 100% F 4 Accounting 100% F 3 Chemistry / Global Resource Systems 100% F 1 Science, transferring into Commerce 25% M 3 Psychology 75% F 4 Kinesiology 100% F 1 Science 100% M 5 Philosophy 50% F 1 Arts One 100% F 3 Human Kinetics 75% Table 11:  Summary of structured real-world connection participant demographics.  The last column of the table gives the students’ score on the CLASS survey Real-World Connections category.  A higher number means the student reports more expert-like beliefs about the relationship between physics and the real world. 5.2.2 Interview Protocol The CLASS survey explicitly probes students’ attitudes towards real-world connections in physics.  However it does not explain why students answer the way they do.  For this study we used the questions from the CLASS “real-world connections” category as a base, but then investigated students’ perspectives more deeply via interview. The students were individually interviewed following a semi-structured protocol. Following the recommendation of diSessa [55] the interviews were conducted in a spirit of mutual inquiry; participants were asked to share their privileged knowledge of how typical students  perceive scientific questions.  The goal of the study was explained as “exploring how you see the connection between science and the real world”.  The interview began with questions about the students’ background and personal history, and then proceeded to giving the four CLASS questions from the Real-World Connections cluster verbally and asking students to answer and explain their response.  These questions are listed previously in Table 7.   78 A particular student’s perception of a particular problem is the result of an interaction between the properties of the problem and that student’s particular experience, personality, and framing of the problem.    The purpose of these interviews was to identify the criteria that students use when making judgments about the connection of problems to the real world and to explore the relative importance of these criteria. In order to support this exploratory aim, the study was structured to solicit the perceptions of a wide variety of students on a wide variety of scientific problems. Students were then presented with a series of problems drawn from real science courses and two abbreviated news stories about scientific issues.  The participants were instructed to read these problems but not to solve them.  These spanned a variety of problem types (short answer, multiple choice, and news story) and science disciplines (mathematics, physics, biology, chemistry, astronomy).  They are listed in full in Appendix C. After the participants had read and considered a particular problem the interviewer asked four questions to probe different ways they might see connections between these problems and the real world.  These questions were developed from a pilot study with two students and are intended to span the different ways in which student in the pilot study expressed different types of real-world connections to scientific information.  These questions, referred to hereafter as Reality Link Questions or RLQs, are listed below. 1. Do you see this problem as connected to the real world?  Why or why not? 2. Do you think that learning more about the topic of this would be useful in your life?  Why or why not? 3. Would thinking about any of your own experiences help you to interpret this question /statement?  Can you give an example? 4. What kind of person might find (the solution to a question like this / this statement) useful?   79 The participants were prompted with these four questions and encouraged to explain their reasoning wherever appropriate.  After a student indicated they were done with a particular question I moved on to ask the next.  Then the student proceeded on to the next science problem, and each of the four Reality Link Questions were asked in order to probe the student’s reaction to that problem.  The interviews were audio-recorded for analysis, and I took field notes during each interview. 5.3 Qualitative Analysis and Results The audiotapes of the interviews were analyzed by both qualitative and quantitative means to investigate which features of the problems the participants saw as connected to the real world. To perform the qualitative analysis I reviewed the audio-recorded interviews to identify overall patterns in students’ responses as well as specific “triggers”: features of the problems that students explicitly cite when justifying their judgments of whether a problem is (or isn’t) connected to the real world.  In addition, any feature that was described as generalizable to other circumstances was counted as a trigger.  Triggers were interpreted based on the single main reason the student gave for or against the connection between a particular problem and the real world. The use of the word “trigger” is rooted in the Framing and Resources perspective of how students make these real-world connections.  When they read a particular problem, certain aspects of the problem may remind them of (or activate) other knowledge that allows them to interpret and contextualize the problem.  For this study I identify problem features that trigger their access to their real-world knowledge, which they employ to judge the relevance of the problem to the real world. I identified five main categories of triggers:  Formalisms in the Problem, Connection to Problem Context, Personal Consequences of Problem, Broad Consequences of Problem, and Implications for Action.    These triggers and overall trends in students’ responses are discussed in the following sections.   80 5.3.1 Judgment Based on Explicit Context The largest trend, which echoed one of the results of the interviews conducted with Physics 100 students after the first year of the course, is that students most often evaluate the relevance of a given problem to the real world based on their personal relationship to the explicit context and question presented.  For example, a question about how Antarctic penguins’ caloric needs are impacted by the insulative properties of their feathers would be most commonly evaluated based on the participants’ own experience with or interest in penguins or the Antarctic, the explicit subject and context of the problem.  However, an expert scientist might evaluate the real-world relevance of that question based on other questions and contexts where the scientific principles in the problem might conceivably apply.  For example, an expert scientist might see that the caloric needs of penguins are important in considering the relationship between a penguin population and their prey, or that the principles of thermal conductivity are applicable to understanding the body warmth of a wide variety of animals, protective cold-weather clothing for humans, or even designing home insulation.  This tendency of students to evaluate real-world relevance based only their personal experience with the explicitly stated context mirrors Chi, Feltovich, and Glaser’s result that students tend to categorize physics problems based on their surface features rather than on deep structure [26]. 5.3.2 Diverse Definitions of Real-World Connections The questions to probe the students’ perceptions of real-world connections from the science problems were specifically crafted to prompt different aspects of these connections. In students’ responses, several themes in the nature of “real-world connections” emerged. Firstly, the interview participants (and sometimes the interviewer) rephrased the term “real-world connections” to mean “real world relevance.” Based on the student responses, I believe that the term “relevant” is actually more specific than the more general “connected”; something that is relevant to the real world is certainly connected to the real world, but something that is connected to the real world is not necessarily relevant. I believe the difference is in whether the problem has meaningful   81 consequences: if a particular problem or issue has consequences but they are not significant or meaningful to the reader, then that problem will not be rated as relevant. As in the more unstructured interviews conducted specifically with Physics 100 students, this diverse group of students continue to distinguish between relevant to other people and relevant to themselves.  I speculate that the latter is more important for motivating learning and meaningful engagement. 5.3.3 Real-World Triggers After coding the transcripts from all ten students the triggers were grouped into five broad categories.  Table 12 below shows the coded triggers and categories.  The number in parentheses after each trigger is the number of times it was coded over the whole data corpus.   82 Objection to Formalisms (12)  -Formulae (2)  -Unrealism of Model (5)  -Calculation (5) Connection to Context (19)  -Animals (2)  -Common/Everyday (2)  -Sports (3)  -Personal Familiarity with Context (12) Personal Consequences (25)  -Personal Consequences (7)  -Money (15)  -Health (3) Broad Consequences (23)  -Societal Consequences (2)  -Environment (21) Implications for Action (12)  -No Consequences for Action (9)  -Inability to Control System (3) Table 12:  Summary of triggers for real-world connections.  The main trigger categories are listed to the left and the subcategories are listed underneath these.  The number in parentheses after each trigger is the number of times it was coded over the whole data corpus.   Triggers that were generally cited as enabling real-world connections are written in bold font, those that were cited as inhibiting real-world connections are written in italics, and those that could be cited as either enabling or inhibiting real-world connections are written in standard font. 5.3.3.1 Examples of Trigger Coding To illustrate the trigger coding scheme two examples are presented below of how transcripts were coded for a trigger.  In the first example, I have been interviewing the student about a problem in which a landscaper needs to do some calculations in order to determine whether it will be cheaper for him to purchase a smaller truck with better gas mileage or a larger truck with a bigger cargo capacity.  The problem is set in the city of Vancouver, and provides all of   83 the specifications of each truck necessary for making a calculation.  The student has already indicated that she thinks that this question is not related to herself, but perhaps to “people who drive trucks”. Interviewer:  Do you think that learning more about this topic would be useful to you in your life? Valerie:  No, I don’t think so. Interviewer:  Why not? Valerie:  Cuz um.. Actually, it might.  Because instead of trucks it could be, like a car and you could be driving people. Interviewer:  OK, so how does that change… How would that… would that be relevant to your life? Valerie:  No, cuz in real life I don’t think we would really be calculating stuff to see which one is a better deal or a cheaper investment. Interviewer:  Mm hm.  So, it’s the calculation? Valerie:  Most people don’t really sit down and do the math to see which one is cheaper.  They probably just go and, like “right there, Ill take that one”  Maybe a rough estimation.  Not, like, actual math. Interviewer:  So what’s the difference between a rough estimation and actual math? Valerie:  I dunno (laughs).  Actual math is when you sit down and write all the numbers, do the actual calculating.  Estimation is when you just do it in your head and round everything off.  In this exchange we see the student begin to generalize from trucks to other types of vehicles, which makes her reconsider her earlier claim that the problem is only relevant to “people who drive trucks”.  However, even in light of this new perspective she remains firm in her opinion that the question is not relevant to the real world, citing her belief that people don’t engage in detailed calculations.  This passage was coded as a “Calculation” trigger. In another transcript, presented below, a different student gives his opinion of the same problem.   84 Interviewer:  Do you feel like this question is connected to real life? Chris:  A hell of a lot more [than the previous question] Yeah, absolutely Interviewer:  Why is that? Chris:  Almost everybody drives.  Personally I don’t, but almost everybody drives.  Almost every business needs to consider fuel costs as a part of their operating overhead.  Even if it’s not necessarily transporting a large amount of goods.  Even if it’s just cars moving small objects from A to B they need to understand the fuel costs, and how much they’re going to spend. And not purely from a financial perspective.  I mean we could use this and the money and turn it into terms of how much gas is necessary to be consumed per day by the area of Vancouver based on the figures given by a certain number of companies that do this sort of thing.  And you’ve got a projection there for how quickly we consume fuel versus how the city needs as a whole.  This is useful to everyone.  Urban planners, business owners.  You live at home, you just need to figure out, well I only drive to and from work.  This allows you to figure out in the long term what the better purchase or lease is for your own personal needs.  In this passage the main reasons this student gives for his perception of the real-world relevance of this problem are that it impacts financial calculations and that driving is a ubiquitous activity.  This passage was coded with both the “Money” and “Common Activity” triggers. 5.3.3.2 Categories of Real-World Triggers Five main categories of triggers emerged from this analysis. BROAD CONSEQUENCES. Many of the problems in this study were deliberately designed to be thematically related to environmental or other broad societal concerns. Environmental consequences emerged as a major category of trigger cited by the interviewees.  In the following example, an interviewee gives his perspective on the real-world connections in a problem about a bird species’ growth under “ideal conditions” on an island with no predators and unlimited access to food.   85 Interviewer:  What do you think about this question? Is it connected to the real world? Bonnie:  Um, it kind of is, I guess. Because, it kind of explains why things are the way they are right now with evolution and that sort of thing, and it also has a little bit to do with the environment and… destroying animals’ habitats and introducing new animals in places and stuff like that. Interviewer:  So, this has to do with destroying animals’ habitats? How so? Bonnie:  I don’t think it has to do with destroying animals habitats, not this question. But it could be applied to that because this is about if you put them on this island and they have all the food in the world and no predators, and that’s kind of the opposite of a lot of the situations we are facing where we’ve introduced unnatural predators that an animal wouldn’t be facing anyway, and we have issues where pollution is killing sources of food, and that’s a pretty big issue right now. So I think this is related to that and could help you better understand that so you kind of know what people are talking about when they’re talking about environmental crises and stuff like that  When the student considers this question, it is the connection to environmental consequences that he believes are “a pretty big issue right now” that form the main real-world connection that he perceives. Because this characteristic is cited so clearly, this passage is coded as an Environment trigger. PERSONAL CONSEQUENCES. Several of the triggers for real-world connections involved personal consequences of the problem being discussed.  While these consequences may not have been present in the problems themselves, some students generalized the substance of the problem to mention the potential for impact on themselves. In some cases, triggers in this category were cited as a reason that a problem did NOT have real-world connection.  In the following example, a student distinguishes between consequences for others and himself and asserts that despite obvious consequences for the environment, he doesn’t feel that a problem about destruction of ozone by CFCs has personal consequences for his life.   86 [Student has just read the problem, which concerns destruction of ozone by CFCs and UV radiation] Ernesto:  Well, I guess I could possibly see this being relevant to the real world but… I guess with the environment.  I wouldn’t really see applying this to real life for me, but I understand that the environment is important and … I mean it’s important that the ozone isn’t affected by CFCs, but I just don’t see this applies to everyday life.  I mean, if the ozone was gone I guess it would apply to everyday life but…  I dunno, I just don’t see it being directly influence to me right now. Interviewer:  It sounds like there’s some conflict there Ernesto:  Well of course there’s this huge thing about the ozone, but I mean.. It’s something that I don’t see ever happening where the ozone will completely disappear, and although it’s important for people to take care of the environment I don’t see how this relates to what’s immediately going on in my life, like immediate concerns rather than overall global concerns.  This example was coded as both an Environment Trigger and a Personal Consequences Trigger.  Ernesto makes an argument for a real-world connection based on its connection to the environment, but against relevance to himself based on lack of personal consequences. CONSEQUENCES FOR ACTION. This category is closely related to personal consequences, but it specifically concerns the implications or possibility for personal actions on the problem topic.  In some cases, students say that because the ideas in or results of a problem under consideration would not have any consequences for action, the problem is not connected to the real world.  A related trigger is when students say that the problem touches on issues that affect them but are beyond their control.  In each case, the lack of implications for students’ choices is cited as a reason for a weak real-world connection.  These triggers are typically cited as reasons that a problem does not have a real-world connection.  Two examples of this trigger are listed below.   87 In the first example, Danielle explicitly says she doesn’t think understanding a particular problem would be useful because it wouldn’t change her behavior.  This is a clear example of lack of personal consequences. Interviewer:  Do you think that learning more about [the problem involving insulation of antarctic penguins] could be useful for you in your life? Danielle:  No, I don’t really think so. Interviewer:  Why not? Danielle:  Because it wouldn’t really change my behavior, whether I knew a lot about this topic or not.  I would still think about things in the same way.  Like if I was going outside and deciding what to wear, like… stuff like body temperature I wouldn’t really think of it differently after learning this stuff.  In the second example, Fiona says that a question about ozone destruction is not relevant to the real world because she has no personal control over the destruction of the ozone.  Therefore this question has no consequences for her actions. Interviewer:  Do you feel like this question is connected to the real world? Fiona:  Like typical real world, no.  But… not really, because we can’t control the UV radiation from the sun.  We can’t control that right?  (Laughs) I don’t think we can control it cuz it’s from the sun, right?  So… I don’t really see how that would help in any way.  It might help in understanding concepts but… helping it, like… helping the ozone destruction.  I dunno.  I did chemistry this summer so, I don’t.. I forgot about it. Interviewer:  Well you don’t have to answer this question, that’s not really what I’m asking.  What I’m asking is, if you see this question do you think that it’s relevant to the real world? Fiona:  Relevant to the real world?  To people who study global warming I guess.  To me, I won’t really care about this question. Interviewer:  Why not? Fiona:  Cuz I won’t really know how to save the world, even if I do know the answer to this, right?  (laughs) Interviewer:  OK, so you won’t care because even if you know the answer to   88 this question… Fiona:  Cuz it’s about ozone destruction right?  So even if I do know the answer it’s not like I can do anything about it. Interviewer:  OK Fiona:  Yeah, even if I’m really advanced in this I won’t be able to do anything about it.  Fiona’s statement that “I won’t be able to do anything about it” is the justification she gives for saying “I won’t really care about this question.”  Her lack of ability to affect the ozone layer (meaning that there are no consequences for her actions) is the reason she gives for her judgment that this problem is not relevant to her. CONNECTION TO CONTEXT. This category of triggers covers situations when students cite specific familiarity with the problem context.  In these cases students typically say that the problem has an obvious real-world connection because of its similarity to their personal experience.  While some of these triggers are quite specific and rare, such as the student who had previously worked with Antarctic penguins during a summer work-study program, many of them are rooted in common everyday experiences such as driving or riding the bus.  These triggers are exactly what we hope for when setting problems in an everyday context. In the following example a student has just read the problem about the landscaper making a buying decision between two trucks with different mileage and cargo capacity (mentioned above).  Without being prompted she volunteers several opinions on the question. Hermione:  I think it’s an extremely important thing to be able to do. This one […] I don’t have to stretch to think of ways that it could be handy.  In fact I was looking at buying a car last year, and that had mileage vs. initial cost vs. all those kind of things was definitely a big thing that I was looking at.  Although in the end I decided not to buy a car at all.   89 Interviewer:  OK.  So would you say that this question is connected to the real world? Hermione:  Definitely.  Because she cites her personal experience in direct connection to the subject of the question, this section is coded as a “Personal Familiarity with Context” Trigger (as well as a “Money” trigger from the Personal Consequences category). FORMALISMS. This category is a collection of negative triggers in which students say a problem is not connected to the real world because of one or more abstractions or mathematical formalisms present in the problem.  Examples include problems that contain formulae, mention extreme simplifying abstractions, or contain calculations in a real-world context where the student believes they are unwarranted.  In the example below, a student gives his perspective on the problem concerning population growth of a species of bird under “ideal conditions” Chris:  [immediately after reading the problem] Um, it’s a useful mechanism for explaining Darwinian evolution.  And again, you have a systems approach and you can abstract out to any kind of predator-prey scenario, or any kind of human habitation.  Once you understand the numbers behind when we go to a certain area, and the resources we consume and what we need. But its still kind of… ideal conditions, no predators, unlimited food is not a real- world situation.  The problem is simplified to the point where it’s no longer… you know it’s not connected to the real world because of those three conditions. Interviewer:  Which three? Chris:  Ideal conditions, no predators, unlimited food [Later in the discussion of the same problem] Interviewer:  So what kind of people do you think would find the answer to this   90 question useful? Chris:  Well no-one, because it’s ridiculously idealized.  I can appreciate that in low-level science you need to simplify the problems as much as possible so people can kind of understand the core concept and then you can start adding complications.  But without those complications the answer isn’t interesting. Because this situation does not exist.  It only exists the complications.  So in and of itself it’s only useful as a tool to gauge whether or not the student understands the core concept.  So I don’t think anyone would find the answer interesting, from that perspective.  While the student does point out the potential value of an idealized question in an educational setting, he clearly believes that the idealizations negate the problem’s connection to the real world.  This segment was coded under the “Unrealism of Model” trigger. 5.3.4 Discussion The perception of the relevance of a particular problem is an interaction of the problem’s characteristics and the student who reads it.  Many of the above-mentioned triggers are clearly related to particular elements of a student’s past history and personal experiences which are difficult for an instructor to control.  This suggests that it may not be possible to create problems that have real-world relevance for most students. 5.4 Quantitative Analysis and Results The importance of students’ idiosyncratic histories and expectations in perception of real-world relevance suggests that perhaps attending to particular problem features as an avenue for promoting the perception of relevance is missing the point.  Indeed, the qualitative analysis makes it clear that these factors are extremely important.  However, it is also clear that some problem factors are important to many students, and it is certainly possible to develop problems that are “better” than others, in the sense that they are more likely to offer students an opportunity to connect their own particular history to the problem context.  For example, problems set entirely on Mars would offer students very   91 little opportunity to do so, whereas problems set in their hometown would be comparatively more likely for students to connect with. In the interest of examining whether there are problem characteristics that broadly increase students’ perception of the connection of a problem to the real world, coding was performed on the problems and student responses and an ANOVA analysis was performed to identify which problem features were significantly correlated to higher overall ratings of relevance. 5.4.1 Coding of Problems The problems were coded for several main characteristics that showed up in the trigger analysis and could be coded objectively.  The problem characteristics that were coded are listed on Table 13.  The complete list of problems can be found in Appendix C. Because of the Calculation and Formulae triggers, the problems were coded as to whether they required a quantitative answer and whether they contained obvious formulae.  To correspond with the Money trigger the problems were coded as to whether they explicitly involved a financial aspect.  Finally, to correspond with the Connection to Context trigger the problems were coded based on whether they were set in an everyday context:  a code of 0 was given to problems with no attempt at a real-world context; 1 was given to problems with a real-world context that was unlikely to have been part of a typical university student’s experience, and 2 was given to problems set in an everyday context.  These codes were independently conducted by two independent researchers with 90% agreement before discussion and 100% agreement after discussion.  Note that in subsequent analysis, these codes were treated as ordinal (rather than a scalar) values.   92 Problem # Qualitative / Quantitative Everyday Context Involves Money Formulae Environmental Rating Landscaping Truck 1 quantitative 2 Y N Emperor Penguins 2 quantitative 1 N N 2 Galapagos Finches 3 qualitative 1 N N 5 Bicycle Commuter 4 quantitative 2 Y N 5 CFCs 5 qualitative 1 N Y 6 Jellyfish 6 qualitative 2 N N 3 Rolling Balls 7 quantitative 0 N N Coriolis 8 quantitative 2 N Y Solar Wind 9 qualitative 1 N N Restaurant 10 qualitative 2 N N Table 13:  Properties of scientific problems used to probe students’ perception of relevance to the real world.  Columns 2-5 were coded by the researcher.  The Environmental rating is based on the number of environmental triggers mentioned by the participants in study. Because of the large number of instances of the Environmental trigger I attempted to code the problems based on degree of connection to environmental issues. However it was difficult to judge the “direct-ness” of environmental connections, and the inter-rater reliability of these codes was only 60%.  Rather than try to code the problems as environmental or not a priori, the interviewees’ own triggers were used to assign an Environmental rating to each problem.  The rating for each problem was the number of triggers (out of ten students) that were coded for that problem. 5.4.2 Coding of Responses to Interview Problems In order to quantify students’ perception of the real-world relevance of each of the test problems, students’ responses to the four Reality Link Questions were coded to assign a value to each response.  The Reality Link Questions are listed in section 5.2.2 above.  Note that students were not asked to explicitly give a numerical answer during the recitations.  I assigned these codes afterwards by reviewing the taped interviews.   93 The first three questions are yes/no questions, and responses were coded on a three point scale for yes, no, and neutral / mixed.  The fourth question asks “What kind of person would care about the answer to this problem?”  Students’ responses were coded according to the degree to which they generalized from the particular context in their answer.  For example, several students responded that the answer to the question about penguins in Antarctica would only be of interest to a penguin researcher.  These responses were coded as a 2.  One student generalized on the context of “a cold place” and replied that the answer to this problem would be of interest to anybody that lived in a cold climate.  This response was coded as a 3. This coding scheme is summarized in Table 14 below. Code Response to RWC Q 1-3 Response to RWC Q 4 1 No Nobody 2 Neutral/Mixed Specialist in context; slight generalization from context 3 Yes Broad generalization from context Table 14:  Coding scheme for responses to Reality Link Questions.  In each case, a code of 1 indicates the student reported that they perceived little or no real-world connection in that category, a response of 3 indicates a strong real-world connection, and a response of 2 is neutral or mixed. In this manner, a code of 1, 2, or 3 was assigned to each of the students’ answers to the four Reality Link Questions corresponding to each of the test problems.  In subsequent analyses, these codes were treated as ordinal rather than scalar data. 5.4.3 Statistical Analysis of Codes Statistical analysis was conducted using the R software package [58] to examine which problem characteristics were correlated with significantly higher results on the above-mentioned interview scores.  The codes for responses to the Reality Link Questions were treated as ordinal (rather than scalar) data, meaning that while a 3 is definitely better than a 2, it can’t be argued that it is precisely 50% better than a 2. Taking means of these data is not meaningful so nonparametric statistical tests were used.  While the responses to the four real-world connection questions are clearly related, the analyses were performed for each question individually to avoid the   94 assumption that a “3” on question 1 is somehow the same as a “3” on question 4. Because the data on these ordinal scales cannot be considered normal, nonparametric statistical tests were used.  Due to lack of time during the interviews problems 9 and 10 were only completed by a single student and were therefore eliminated from the analysis. Based on the student interviews it was assumed that the Money trigger would increase the results, so this was examined with a 1-tailed Mann-Whitney test to see if it made a significant difference to the results.  Similarly, it was assumed that the presence of formulae in a problem would decrease the real-world connection ratings so this was also examined with a 1-tailed Mann-Whitney test.  Due to some mixed comments on the connection between calculation and everyday life, the Quantitative factor was examined with a 2-tailed Mann-Whitney test. The Environmental code and the Everyday Context code were both examined with two tests:  a 1-tailed Spearman Rho test examined the probability of a nonzero correlation between the ordinal code and the results.  However due to the low number of problems a second test was conducted to increase the power of this test: the Everyday Context analysis was used to split the problems into two groups:  Weak Everyday Context (coded 0 or 1, N=5 problems) and Strong Everyday Context (coded 2, N=2 problems) and a Mann-Whitney test was conducted to see if there were any significant differences between the results of these two groups.  Similarly the Environmental code was used to split the problems up into two groups:  weak Environmental (coded 0-4, N=5 problems) and strong Environmental (coded 5 or higher, N=3 problems). The results of these initial tests are summarized on table 15 below.   95  Test P-value Problem Characteristic RWC Q1 RWC Q2 RWC Q3 RWC Q4 Money 0.006*** 0.005*** 0.017** 0.001*** Formulae 0.971 0.920 0.959 0.800 Quantitative 0.139 0.800 0.072* 0.309 Everyday Context - Spearman 0.108 0.047** 0.087* 0.003*** Everyday Context (2 groups) 0.054* 0.030** 0.034** 0.002*** Environmental - Spearman 0.631 0.083* 0.565 0.265 Environmental (2 groups) 0.251 0.014** 0.404 0.244 Table 15:  P-values of tests to examine the relationship between problem characteristics on student responses to Reality Link Questions.  *(p < 0.1), **(p< 0.05), ***(p < 0.01). According to these tests the problem characteristics with the most significant correlation with the real-world question results was the Money characteristic, which showed a significant difference in all four real-world connection questions (p<0.05) . The Everyday Context rating also showed a strong relationship with the results.  The spearman rho test showed a significant correlation for Reality Link Question 2 (rho=0.23, p<0.05) and Reality Link Question 4 (rho=0.37, p<0.05).  The follow-up Mann-Whitney test on the results of high vs. low Everyday Context showed a significant difference with p < 0.05 in three of four questions, and a difference with p<0.1 in the fourth question.  The Environmental rating and the Quantitative characteristic also showed a significant difference (p<0.1) in one of the four questions. Because of the strong results for Everyday Context and Money, an analysis of the interactions between these two characteristics was conducted.  The problems were separated into three groups:  Group 1 was Strong Everyday Context and Money; Group 2 was Strong Everyday Context and No Money; Group 3 was Weak Everyday Context and No Money.  (There were no problems in the study with Weak Everyday Context and Money, so the fourth possibility was not tested).  A Kruskal-Wallis test of the relationship between Group number and Reality Link Question results showed that at least one of the three groups had significantly different results for RWC questions 1, 3, and 4 (p<0.05).  Subsequent pairwise Mann-Whitney comparison by   96 group was used as a post hoc analysis to see which groups had better results.  These results are summarized in Table 16 and 17 below.  Test P-value Everyday Context & Money Interaction RWC Q1 RWC Q2 RWC Q3 RWC Q4 0.038** 0.035** 0.103 0.006*** Table 16:  P-values of Kruskal-Wallis test to examine the impact of the interaction between Everyday Context and Money on real-world-connection questions.   *(p < 0.1),  **(p< 0.05),  ***(p < 0.01).  Test P-value  RWC Q1 RWC Q2 RWC Q3 RWC Q4 Group 1 > Group 2 0.012** 0.034** 0.133 0.067* Group 1 > Group 3 0.010** 0.007*** 0.019** 0.001*** Group 2 > Group 3 0.550 0.478 0.429 0.172 Table 17:  P-values of pairwise Mann-Whitney comparisons of average results between the three groups of problems.  Group 1 is Strong Everyday Context and Money; Group 2 is Strong Everyday Context and No Money; Group 3 is Weak Everyday Context and No Money.    *(p < 0.1), **(p< 0.05), ***(p < 0.01). These results show that Group 1 (Strong Everyday Context and Money) had higher results in RWC questions 1 and 2 than both other groups, and had higher results than Group 3 for RWC questions 3 and 4  (p<0.05).  This indicates that problems that have both an everyday context and have financial implications are significantly more likely than problems with only one of these to be rated as connected to the real world. 5.4.4 Discussion While Money and Everyday Context were coded as independent problem characteristics, the small number of problems and the small number of participants in this study mean that they may not be measured independently.  It is certainly plausible that the problems that involve Money happen to be the ones that are most deeply embedded in Everyday Context.  However, the combination of the two factors did emerge as a significant indicator that students would rate the problem highly on the Reality Link Questions.   97 The fact that only two of the triggers demonstrated a significant correlation with scores on the Reality Link Questions doesn’t mean that the remaining triggers are not important:  the small number of questions studied and small number of students limit the statistical power of the tests, so it’s very possible that these other triggers could make a difference if studied with a larger sample size. The fact that there is some agreement between the qualitative analysis and quantitative analysis of interview data gives us confidence that some characteristics of scientific problems do have an impact on students’ perception of their connection to the real world.  While not all the triggers highlighted in the qualitative analysis were found to be statistically significant in the quantitative analysis, the low fidelity of the coding and the low number of students in the study limited the power of these tests. A larger study using Likert scales to probe students perceptions of particular problems would enable a better statistical test of the impact of these various problem characteristics. 5.5 Summary Qualitative analysis of students’ comments on a wide variety of scientific problems identified five major categories of triggers that students cite as justification for their judgment of a problem’s connection to the real world.  The categories of triggers are: Broad Consequences; Personal Consequences; Consequences For Action; Connection to Context; and Formalisms. Quantitative analysis identified two specific triggers that had a significant effect on students’ overall rating of the relevance of a problem to the real world.  When a problem was set in an Everyday Context and contained a Monetary motivation it generated higher ratings of relevance to the real world than problems that had only one or neither of these characteristics.   98 6 EPISTEMOLOGICAL FRAMING AND REAL-WORLD CONNECTIONS IN STRUCTURED GROUP PROBLEM-SOLVING 6.1 Introduction Many teachers, researchers, and policy makers have described the importance of connecting science education to real-world phenomena. Recently there has been an increased focus on scientific literacy, which implies that students must not only learn the connections between science education and real life, but must also learn to make use of their science education in order to understand and act in the world [59]. To support the development of their students’ scientific literacy the transformation of Physics 100 in 2007 had an explicit goal of offering students an education in physics that they would be able to apply outside of the classroom.  In order to achieve this the course supported development of students’ skill at solving complex real-world problems.  As well, we tried to provide opportunities for students to integrate their formal physics knowledge and their real-world knowledge by encouraging them to use their real-world knowledge within the physics classroom. The Physics 100 recitations were designed to support both of these goals.  They are based on research-based methods for promoting development of expert-like problem- solving skills, and are organized around a structured problem-solving strategy that is intended to promote qualitative and conceptual discussion at appropriate times during the solution process.  As well, they are set in an everyday context and are designed to enable students to make Real-World Connections by using their own knowledge within the context of the physics course. This chapter focuses on examining the effectiveness of structured problem solving steps at promoting the use of conceptual and qualitative knowledge during problem solving and on the general characteristics of students’ Real-World Connections in problem solving.  In the following sections I will describe my research questions, present my theoretical framework, introduce my coding schemes for analyzing students’ Real-World Connections and epistemological framing, and then discuss the observed relationships between structured problem-solving, framing, and Real-World Connections.   99 6.1.1 Research Questions The study presented in this chapter addresses the following research questions. 1. In what ways do students make use of their real-world knowledge during collaborative group problem-solving?  This question examines the notion of students’ connections between formal physics and the real world within the specific context of their weekly recitation sessions.  Students’ use of real-world knowledge is examined via qualitative review of their discourse during group problem-solving to identify categories of ways in which students use their everyday knowledge within the recitation context.  These instances of students’ use of everyday knowledge are defined as Real-World Connections, and a robust coding scheme is developed to identify them. 2. How does students' framing affect whether and how they make use of their real-world knowledge during collaborative group problem-solving?  This question is the first of two that examine different factors that promote students’ Real-World Connections.  It is addressed by coding recordings of students engaged in group problem solving and examining how their Real-World Connections correlate with their epistemological framing. 3. How does the structured problem-solving strategy affect whether they make use of their real-world knowledge during collaborative group problem-solving?  This question is the second of two that examine different factors that promote students’ Real-World Connections. This question is addressed by coding a variety of recordings of students engaged in group problem solving and examining how their Real-World Connections correlate with the structured problem-solving method used in the recitation. 4. To what degree are structured problem-solving methods effective at promoting the use of conceptual and qualitative knowledge at the intended times in the solution process? This question is addressed by coding a variety of recordings of students engaged in group problem solving and examining whether they engage in conceptual discussion when prompted by the structured problem-solving prompts written on their recitation worksheet.   100 6.1.2 Research Context The Physics 100 recitations were developed in order to support the two course goals of: 1. enabling students to learn how to apply physics to novel problems outside the classroom and 2. encouraging students to see physics as relevant to themselves and to their lives. Context-rich problems were chosen for the Physics 100 recitations based on Heller et al.’s result that students working cooperatively to solve context-rich problems exhibited more expert-like problem solving [1].  These problems also offered students an opportunity to practice making simplifying assumptions, an activity that is essential for solving real-world problems but is not developed in conventional educational settings [48]. Initially, we used problems that were directly drawn from the Minnesota online archive of context-rich problems [60].  Based on guidelines developed from my study of students’ perception of real-world connections in science problems (described in chapter 5 above), new problems were subsequently developed that were set in an everyday circumstance, motivated by a plausible reason for calculation, and yielded a result that has clear consequences.  The intent was to offer students opportunities to make use of their everyday knowledge within the recitation context and to show them circumstances where physics was relevant to achieving a goal that they found realistic and relevant. As described in Section 2.4.4 above, the Physics 100 recitations used a structured problem-solving strategy.  The same series prompts for each step in the strategy was written on the worksheets each week with an appropriate amount of whitespace underneath. In order to address the learning curve for dealing with an unfamiliar problem- solving method, the first five recitations were workshops focusing on each of the problem solving steps, similar to the “skills progression” approach advocated by   101 Teodorescu [20].  This workshop structure echoes Van Heuvelen’s recommendation to “provide students with explicit instruction in the individual skills used by experienced physicists when solving complex problems and then help them combine these skills to solve complex problems” [21].  As much as possible, these workshop problems were also contextualized in an everyday context.  After the five introductory workshops the students were required to use the complete problem-solving method to address a single context-rich problem each week. Students worked in groups of 3 or 4, and each group was given one worksheet to complete.  During the year of this study, students were assigned into groups according to the guidelines described in Section 2.4.5.2 above.  The students were also required to make use of rotating group roles each week, but informal observations and feedback from the course indicated that these roles were not taken seriously by the students. 6.2 Background and Theoretical Framework 6.2.1 Resources and Framing One of the key things I study is students’ use of their real-world knowledge within the physics context.  During the first few years of the course’s implementation I noticed that even in situations where students had real-world knowledge that was relevant to the physics situation at hand they did not always employ it. Often, use of knowledge from one domain in another is treated as a purely cognitive task, and is considered from the perspective of transfer.  However the notion of transfer does not help us to understand why a student may blithely report an unrealistic and nonsensical answer for a physics problem and then immediately realize that it is nonsensical as soon as she is asked to consider the realism of her answer.  It seems implausible that her knowledge of the real world and her capability of making connections between the calculated result and her experience has changed quickly enough to explain this phenomenon. Instead of considering this phenomenon as an example of transfer I use the theoretical framework of Resources and Framing to help understand why a student   102 might not make use of her common sense in one moment but then be able to in the next moment.   This theoretical framework proposes a structure whereby a particular frame can activate or inhibit certain resources [28,61,62].  As I will explain below, it is the change in a students’ frame that explains why they may be able to access their real-world knowledge one moment and not the next. Resources are fine-grained elements of knowledge that may be accessed as a part of the process of thinking about something.  The kinds of resources that I am most interested in are cognitive (having to do with declarative knowledge or processes) and epistemological (having to do with the nature of knowledge and its construction). Resources are connected to each other and activation of a particular resource can excite or inhibit the activation of another resource.  Through networks of mutual activation and reinforcement, resources are commonly activated in networks. The notion of a frame helps us to describe overall patterns in these activations, and to connect the notion of resources to linguistic analyses [63].  A frame is a person's implicit sense of the essential nature of the activity that they are engaged in. It is their answer, which is often subconscious or implicit, to the question “what's going on here?”  The implicit sense of the nature of knowledge and learning that a person is using in their present moment is called their Epistemological Frame.  A person’s epistemological frame can govern which knowledge is valued, what is regarded as “the right answer”, and which strategies are employed in order to get it. While a person’s epistemological frame is usually consistent from moment to moment, it can change rapidly in response to new information, interaction with conversational partners, or other prompts. The theory of Resources and Framing helps us to understand the example given above.  If a student initially frames her activity as working on a physics problem, she may take that to entail that only physics knowledge is relevant.  This might lead her to unquestioningly trust the results of mathematical calculation. That same framing of working on a physics problem may lead her to hold the implicit belief that her own knowledge about the real world circumstance is not valid or valuable, and so she might never think to compare the mathematical answer to her own knowledge of the   103 physical situation. She treats the problem as a puzzle or exercise rather than as an investigation into a model of the real world which has a meaningful connection to real events.  An instructor’s inquiry or suggestion can trigger her to change to a new frame where she treats her result as representative of a real quantity, upon which point her common sense about the real world may be activated and brought to bear. A change in framing grants the students a new perspective, and allows her to access her everyday knowledge. 6.2.1.1 Review of Scherr and Hammer Coding Scheme To identify students’ framing during group problem solving I use a coding scheme that is based on one developed by Scherr and Hammer [64].  In this section I review Scherr and Hammer’s coding scheme as presented in their 2008 paper. Scherr and Hammer’s key claim is that “behavioral clusters” - groups of vocal and bodily gestures that tend to co-occur - are evidence of and mutually interact with student epistemologies.  They describe four stable epistemological frames that are evidenced by student behavioral clusters, are shared by all group members, and correspond to patterns in the group’s approach to knowledge and learning. By conducting detailed analysis of patterns of students’ behavior as well as the thinking that is inferred from their speech, Scherr and Hammer showed connections between the substance of individual students’ thinking and the nature of behavioral interactions among members of the group.  They concluded that verbal and nonverbal displays reinforce implicit messages about an individual’s epistemological framing, contributing to the participants’ mutual understanding of what is taking place.  These meta-messages carried in behavioral cues help groups to construct their mutual epistemological frame and coordinate shifts between frames. Their analysis of students’ framing during collaborative problem solving was conducted on videotapes of groups of four students working on specially-   104 constructed tutorials that explicitly solicit and address students’ everyday knowledge and epistemological resources [65].  Each student completes their own worksheet but they are frequently instructed to discuss ideas with each other.  The groups studied were selected based on the fact they were consistently on task and tended to talk to each other frequently. Based on their analysis, Scherr and Hammer identified four main clusters of behavior which correlated with particular practices with regards to developing and validating the knowledge under consideration.  Based on the mutual coherence of these behaviors and epistemological practices, Scherr and Hammer argued that the four behavioral clusters correspond to four distinct epistemological frames.  The four frames from their scheme are listed on Table 18 below. Blue:  Worksheet Frame Green: Discussion Frame Behaviour Expectation Behaviour Expectation Hands quiet, face neutral Body leans forward, eyes on paper Brief glances at peers Muttering Minimal interaction, individual activity Attention belongs on the worksheet “Check-ins” expected Peers not attending to details of speech Prolific gesturing Animated tone, face Sit up straight, eye contact Clear utterances Peers are watching and want to understand Intellectual and/or emotional engagement Attention belongs on peers Peer interest in details of speech Red:  TA Frame Yellow:  Joking Frame Behaviour Expectation Behaviour Expectation Sit up straight, eye contact with TA Reduced gestures Attention belongs on TA Rehashing thinking Giggle, smile, self- touch, fidget, unsettled gaze Embarrassment, perceived vulnerability Table 18:  Four behavioral clusters and associated epistemological frames.  Scherr and Hammer argue that these behaviors in group problem-solving constitute evidence for students’ current approach towards knowledge and learning.   105 The four frames described by Scherr and Hammer are: 1. the Worksheet frame, where students’ attention is on the worksheet and their implicit understanding of the task is completing the worksheet task 2. the Discussion frame, where their attention is on each other and their implicit understanding is that they are discussing each others’ ideas 3. the TA frame where their attention is on the TA and they understand their task to be paying attention to the TA 4. the Joking frame where their attention is unsettled and they understand themselves to be joking around. The bulk of the students’ meaningful engagement with the physics content occurs in the Discussion and the Worksheet frames.  In the Discussion frame sitting up, speaking clearly, and gesturing frequently appears with novel reasoning and mutually-constructed understanding.  Scherr and Hammer state that in the Discussion frame “the substance of students’ conversation is the physical events under consideration.” Conversely, in the Worksheet frame students’ attention is on their worksheets and their body language and speech display an expectation that they will not be meaningfully communicating with each other.  There is often some speech in this frame however; Scherr and Hammer say that students in the Worksheet frame “do often speak to one another, typically reading from the worksheet, giving brief status reports of their progress, and requesting or providing information”.  They state that these interactions are primarily to convey information in support of completing the worksheet. During the recitation, student groups will transition between these different frames as the focus of their interaction shifts.  A group’s frame can shift in response to any external cue or in response to a bid from one of its members:. Scherr and Hammer argue that behaviour by one student that does not fit with the current behavioral cluster is often an implicit bid for the group to change to   106 a new frame.  Sometimes bids are ignored, and the rest of the group continues in the current frame.  At other times another group member will take up the bid by responding in the new frame, effecting a shift of the whole group’s focus over to a new frame. Scherr and Hammer note that analysis of framing can be conducted at different scales. If one is focused on the individual cognitive perspective, one can conceptualize frames as being a property of the individual, and communication among individuals in a collaborative group serves to communicate about their own framing and coordinate frame transitions among the group. However, one could also take the perspective that the frame is a property of the group itself, and all of the speech at and body language of the group are evidence of “the group’s frame”. For my work, I will take more of the former view, although I will assume unless there is evidence of the contrary that a group tends to share the same frame. 6.2.2 Real-World Connections 6.2.2.1 Definition of Real World Connection Some physicists might object to the notion of “real-world connections” as being somehow distinct from the normal study of physics. For many physicists, the study of physics is the study of the real world. However research has shown us that the belief in the deep correspondence between physics and the real world is a belief most commonly held by experts and students do not always see things in that way. In the development of surveys of students' beliefs about physics, both Redish and Adams have noted the tendency for novices to treat physics as being separate from the real world [13,54]. The resources and framing theoretical framework offers us a way of operationalizing the notion of real-world connection. I assume that the resources that students develop in physics class are heavily associated with performance in physics class. When addressing a problem posed in the   107 specialized language of physics problems in the context of a physics class, students are more likely to address that problem by making use of the things they learned in that physics class.  In the same way, students’ everyday knowledge is developed outside the classroom and is more likely to be activated in those contexts.  Within this framework, I define a Real-World Connection as an instance when students cross these boundaries and make use of physics resources in their everyday lives or make use of everyday resources in a physics context. Note that this definition implies a separation of the world into two spaces: physics class and everywhere else, and I refer to the resources rooted in those two spaces as physics resources and everyday resources.  As mentioned above, this distinction is not intended to imply that physics class is somehow not a part of students’ everyday lives, but rather to reflect the separation between the different kinds of knowledge that students commonly perceive. For the purposes of this study, I will be looking for instances where students make use of their everyday resources in the context of a group physics problem- solving session.  The details of my methodology and rationale for our approach will be described in the following sections.   However first I will discuss why these real-world connections are important to promote and study and then discuss some of the prior research that attempts to investigate real-world connections. 6.2.2.2 The Importance of Studying RWC in Physics There are many reasons why science educators and researchers care about making connections between physics and the real world.  Principally, making connections between formal physics and everyday knowledge is what makes it possible for novices to learn physics.  In order to forge an understanding of new and unfamiliar language, concepts, and representations, a learner must meaningfully integrate these into their existing framework of knowledge and experience [2].  For someone who is not yet an advanced physicist, this means   108 they must learn physics by interpreting it in terms of their existing conceptions about how things work in everyday life.  There is also evidence that students’ connections between science and their everyday lives are correlated with improved learning and retention.   Pugh et al. also found that students who make meaningful real-world connections are more likely to succeed at a delayed assessment and at a transfer task, and suggests that “as students apply the concepts they learn in the classroom to their everyday lives […] they become more fluid and agile in thinking about these conceptions, thus increasing their transfer ability” [7]. Making real-world connections in physics can also have a strong impact on students’ perception of the relevance of physics to themselves, with a concomitant impact on their persistence in the field. Osborne and Dillon [66] have found that a lack of perceived relevance was one of the reasons for students’ lack of engagement in science.  Hazari, Sonnert, Sadler, and Shanahan conversely found in a study of 3829 students that frequency of connections to everyday life in high school physics was significantly correlated with higher identification with physics and an increase in planning a career in physics [67].  If we can enable students to see how physics helps them to understand or to solve problems that they care about in their everyday lives, they are more motivated to learn and pursue physics. Hazari et al. also showed that actively promoting real-world connections in physics may be important for promoting science identity for females in particular.  In their study, females were significantly less likely to report that their high school physics class discussed currently relevant science topics, despite the fact the males and females in her study were in the same classrooms.  This suggests that the science topics discussed were less relevant to the females, demonstrating the need and opportunity for identifying science that would be relevant to the females as well.   109 Another argument for real-world connections in physics can be made from the perspective of long-term scientific literacy. The recent National Science Education Standards emphasized scientific literacy as a key goal of science education, and understanding of connections between physics and real world issues is a key component of scientific literacy [68].  Understanding the connection between cultural, political, or social issues and scientific principles helps to inform people’s engagement with those issues and with the recommendations of scientists. Considering these impacts on learning, transfer, retention, persistence, identity, and scientific literacy, it is unsurprising that researchers on students’ beliefs about physics have identified a belief in the intimate connection between physics and the real world as a key element of an expert-like attitude towards physics [13,54].  Learning to appreciate and make use of this relationship is one of the hallmarks of an expert physicist, and if we hope for our students to develop this expertise we must help them to learn to see the world and physics in this way.  Studies using these same surveys of student beliefs shows a correlation between scores on these surveys and content learning, supporting the notion that helping students to connect between physics and the real world helps them to learn physics. Within the Physics 100 Tutorials, I assume that if students are able to make use of their real-world knowledge within a physics class that this will have two benefits. Firstly I assume that the practice of using their everyday knowledge in conjunction with formal physics knowledge in a physics context constitutes a rehearsal for the practice of making use of these two kinds of knowledge in the everyday world, and will thereby improve students’ ability to apply physics knowledge outside the classroom.  Secondly, I assume that making productive use of everyday knowledge in a physics context demonstrates to students the deep connection between physics and the everyday world, which will change their declared and implicit belief in the relevance of physics to the real world. These beliefs are important because they impact whether a student will even   110 think to make use of their formal physics knowledge in an everyday context.  As Pugh, in his research on whether students make use of science outside the classroom, has said, “The ability to apply knowledge does not guarantee that students actually will apply their knowledge” [8]. I do not necessarily argue that each instance of a real-world connection will necessarily impact the students’ learning and beliefs about physics but rather that each is an opportunity for such an impact.  In a similar manner to Sawtelle [69], who conducted moment-by-moment analysis to identify opportunities for students to develop self-efficacy, I argue that attending to and encouraging these real-world connections will create more opportunities to affect students’ learning and beliefs about physics and will have the effect over time of achieving the desired improvements. 6.2.2.3 Previous Work on Examining Students' RWC Making use of my earlier definition of Real-World Connections as students’ use of physics resources in the real world or vice versa, we can identify many other studies that have looked at students’ real-world connections from different perspectives.  In this section I will describe some of these studies and some of their limitations that will be addressed in my thesis. One method of examining students’ ability to apply scientific knowledge in the real world is by testing their performance on science questions that are contextualized in the real world [23,24].  In both of these studies, the researchers gave the students a test which included questions set in the real world where physics knowledge could be productively applied, and based on the results of those tests made claims about students’ abilities to make use of physics knowledge in a real world context.  However these studies share the important limitation that students are likely to frame a paper-and-pencil test quite differently from an authentic engagement with a problem in the real world.  As such, it is likely that the resources they apply to the test are substantially different from those they would apply in the real world.   111 Another approach to examining students’ use of science in their everyday lives is Pugh’s research on Transformative Experiences, which are defined as an instance of spontaneous use of science concepts in everyday experience in a way that changes the way a student sees the world [8].  He argues that students may engage in a variety of transformative experiences, ranging from meaningful engagement in the classroom on one end to intentionally seeking out examples of science concepts in their everyday lives on the other end [7].  This perspective is interesting in that it highlights the idea that a real-world connection need not occur in the physical context outside the classroom, but can be the product of students thinking about scientific and everyday ideas together. Pugh’s research does not attend to the use of everyday knowledge in a scientific classroom.  One limitation of this is that this research relies on measures of students’ transformative experiences via their responses to survey questions about what they do, think, and believe during their everyday lives. Such a survey is vulnerable to errors of poor student recollection, and is likely to record how students believe they act (or believe they ought to act) rather than their actual deeds, thoughts, and beliefs in that context. One study which looks at students’ actual behavior rather than their test results or survey responses is Mayoh and Knutton who looked at the use of “out-of-school experience in science lessons” in 103 science lessons in British comprehensive schools [25].  This study looked for the use of “any experience or understanding arising outside formal classroom-based instruction” within students’ normal science classrooms.  They examined both teachers’ and students’ references to the world outside the classroom, and compiled a taxonomy of different kinds of episodes where students or teachers mention out- of-school experiences during science lessons.  While this research is illuminating, it was conducted in classrooms where teacher-mediated discourse was the norm: 69% of the episodes coded as involving out-of-school experiences had the teacher as one of the participants.  Because I assume that it   112 is the students’ own connections that are meaningful to their learning and motivation, and because I have found that teachers’ and researchers’ ideas of what ought to be a real-world connection do not necessarily impact the students, I argue for a closer focus on the students. Another thread of research that is relevant to the study is the wide variety of research into students’ intuitive understandings of science and the ways that they come into play in learning environments that has been conducted.  (See [3] for an extensive list.)  This research is motivated by the result that students’ intuitive knowledge and prior understandings impact whether and how they learn formal science content. While there are exceptions, this research is largely focused on how students’ real-world knowledge and intuitions affects their physics learning, and does not illuminate real-world connections that may impact students’ attitudes or identity.  In order to address these dimensions of students’ experience in physics, a broader research focus is needed. 6.2.2.4 My Approach The approach used in this study eschews the analysis of tests and surveys, and is instead focused on students’ discourse and behavior as they work collaboratively on problems where real-world resources can be productively applied.  I try to include and identify Real-World Connections that are affective as well as cognitive, as I am interested in examining real-world connections that affect students’ identification with physics and their judgment of its relevance as well as their learning.  The details of my methodology for coding students’ Real- World Connections are described in section 6.3.5 below. 6.2.3 Conceptual Discussion During Problem-Solving The problem-solving strategy used in the Physics 100 recitations includes several steps that are explicitly intended to induce students to make use of their conceptual and qualitative knowledge both prior to and following their main calculations. Several published problem-solving strategies also include steps with the same intention [1,20,21].   113 According to Heller et al., context-rich problems were designed to “focus students’ attention on the need to use their conceptual knowledge of physics to qualitatively analyze a problem before beginning to manipulate equations” (p.629; All quotes in this paragraph from reference [1]).  The structured problem-solving strategy that is recommended for use with these problems was designed to help students “integrate conceptual and procedural aspects of problem solving” (p. 628).  This strategy requires students to “make a systematic series of translations of the problem into different representations, each in more abstract and mathematical detail” (p.628). For example, step 2 requires students to “use their qualitative understanding of physics concepts and principles to analyze and represent the problem in physics terms,” (p.629) explicitly directing students to perform a translation of representation from the narrative of the problem story into more formal physics abstractions. The form of this strategy is informed by studies which focused on differences between how experts and novices solve problems [26,27].  A key finding of the expert-novice research is that experts are more likely to make use of qualitative reasoning before and after employing mathematical methods in solving physics problems.  This ability to make use of qualitative knowledge has been identified as a crucial asset in successfully solving complex problems [70].  Experts were also seen to make more frequent use of specialized representations when solving problems.  These results have motivated Heller and others to structure their problem-solving strategy around such translations of representation [20,21] and to explicitly encourage the use of qualitative knowledge before and after mathematical solutions.  Many strategies enforce adherence to the expert-like strategy by requiring students to perform each step via a marking rubric or worksheet [1,20,21]. The use of such a strategy and the requirement that students follow each step implicitly assumes that the students will, in some meaningful way, follow the instructions laid out in each step.  In the case of the above strategies, the requirement to develop an expert-like sequence of representations is intended to induce the students to perform the required tasks and presumably get better at developing representations and integrating conceptual knowledge by doing so.   114 Some research has borne out the merit of this approach.  In a clinical setting, Dufresne et al. demonstrated that the use of a step-by-step computer guide to constrain novices to engage an expert like problem-solving behavior produced improved problem-solving performance [71]. However Heckler, working in a classroom setting, saw decreased performance as a result of explicitly prompting students to produce a free body diagram prior to solving a dynamics problem and suggests that this prompt “cued some students to the mindset that constructing the diagram and solving the problem are two separate tasks” [72].  His suggestion is that the use of prescribed problem-solving prompts may prime students to treat problem-solving steps as a list of instructions to follow rather than as individual elements that contribute to an overall understanding and coherent problem solution.  In addition, Taconis’ meta-analysis of research on problem-solving methods concluded that “attention to knowledge of strategy and the practice of problem solving turned out to have little effect [on students’ acquisition of expert-like problem- solving skills],” suggesting that overall these structured methods may not be helpful in improving problem-solving performance [73].  Considering that structured problem- solving strategies are featured in many leading textbooks (e.g. [74,75]), the prospect that they may be ineffective is troubling. In this study I do not examine the long-term effectiveness of these strategies at improving problem-solving performance or promoting development of problem- solving skills, but instead focus, via audio-recordings of their conversations, on students’ immediate interactions with the printed prompts that scaffold the prescribed strategy as they work through a problem.  Given that these strategies are intended to induce students to make use of their conceptual knowledge at appropriate times during the problem-solving process, I examine students’ discussions to investigate whether these strategies are successful at doing so.  In particular, I identify an epistemological frame where students’ discussions are aimed towards figuring out the meaning of the ideas under consideration, and examine each of the problem-solving steps to see when students engage in this frame.   115 6.3 Methodology 6.3.1 Overall Analysis Strategy In order to address the above research questions I examine students’ discourse and behavior during their normal weekly recitation session and code them for three different types of information:  instances when they encounter particular structured problem solving prompts on their worksheets, epistemological framing, and instances when they are making Real-World Connections.  Coding is conducted by reviewing audio recordings of students’ discussion during recitation sessions, transcripts of these recordings, and (for approximately half of the episodes) field notes I took while sitting with the students as they work.  In my analysis I attend to both the text and meaning of students’ speech as well as their turn-taking behavior, rhythm, and prosody of speech, after the tradition of conversation analysis [76, 77].  These characteristics offer insight into the substance of students’ ideas as well as into the ongoing negotiation of their framing of their activity [78, 79]. These codes are subsequently examined to reveal correlations between epistemological framing, problem-solving prompts, and Real-World Connections and to address the research questions.  Figure 11 below serves as a map for the coding and analysis in this chapter and provides a reference to help locate the details of each coding scheme and correlation study.   116  Figure 11:  Main coding schemes and correlation studies in Chapter 6. Below I briefly discuss the role of Epistemological Framing in this analysis and describe the cohort and recruitment. Subsequently, I describe the three coding schemes in detail and illustrate these codes and correlations with particular examples of student dialogue.  Then I describe the correlations observed between these codes and use analysis of selected episodes to develop and support causal arguments around the observed patterns of correlation.   Correlation Between Frame and RWC Section 6.4.2  Correlation Between Prompts and CD Section 6.4.3  Correlation Between Prompts and RWC Section 6.4.2 Problem- Solving Prompts Epistemological Framing Real-World Connections Background Summary of Coding Categories 0. Begin 1. Interpret the Problem 2. Identify Relevant Physics 3. Model: Assumptions 4. Model:  Diagram 5. Solve 6. Error-Checking and Sensemaking Section 6.3.4 Section 6.3.5 Section 6.3.2 Section 2.4.4 Conceptual Discussion (CD) Procedural Discussion (PD) Worksheet (W) TA Focus (TA) Group (G) Meta-Comment (M) Other/Off-topic (O) Section 6.3.4.5 Explicitly negative about realism of tutorial (Neg) Not explicitly negative about realism of tutorial (Pos) NB: Negative RWC are analyzed in Section 6.4.5 Section 6.3.5.3 Coding Scheme   117 6.3.1.1 On the Role of Epistemological Framing As described in section 6.2.1 above, a person’s epistemological framing is their implicit sense of the nature of the activity they are engaged in with respect to knowledge and learning.  In examining the different research questions in this study, the construct of framing is treated somewhat differently in different parts of the study. In my examination of the Physics 100 recitations I will pay close attention to how students are framing their engagement with the problems in order to reveal how this framing correlates with their use of real-world knowledge.   The Resources and Framing framework tells us that frames can increase or decrease the likelihood of activation of different sets of resources.  Because I define Real- World Connections as activations of resources rooted in students’ lives outside the classroom, I consider their frames as being the structure that controls the likelihood of real-world connections.  While recognizing that a real-world connection can induce a frame shift, for the purposes of this study I treat the frames as influencing Real-World Connections. In my examination of the impact of problem solving prompts on the use of conceptual and qualitative information I will use epistemological frames to measure whether students are engaged in conceptual discussion with each other.  In that study I treat the frames as being influenced by the structured problem-solving prompts. 6.3.1.2 Cohort To examine students’ Real-World Connections in a typical physics context and their interaction with particular recitation features, this study focuses on in situ observations of groups of students collaborating in their regular weekly recitation sessions. The group nature of the recitation work requires the students to explain and negotiate with each other which helps to reveal their students’ thoughts and reactions to the recitation.  As well, these peer interactions reveal their framing as they make use of meta-messages to coordinate their activity.   118 In an effort to get a cohort that represented the diversity of the student population, 14 different student groups were observed; one group per episode. The cohort for these studies was recruited in several different ways.  Some of the student groups volunteered in response to in-class announcements. Because the groups that volunteered disproportionately represent the students that had high CLASS survey pre-scores, I was concerned that these students might tend to approach their recitations differently and therefore introduce a selection bias into the study. In an effort to solicit broader representation of the student population, some groups were recruited by directly asking for participants during the recitation sessions.  I would attend the recitation session, and before the normal introduction to the session would explain to the students that I was an education researcher and soliciting participants for a study in order to better understand how students react to the new recitations.  I explained that participation was optional, anonymous, and would not impact their grades in any way and they would not be required to do anything out of the ordinary, nor would they be compensated for their participation.  Then, while the TAs were giving their normal introduction to the recitation, I would look in the room to identify student groups that appeared attentive and engaged in the TAs speech and for student groups that appeared disinterested and/or disengaged.  I would approach one group from each category after the TAs’ introduction and invite them to participate in the study.  If any group members appeared uneasy or refused I would excuse myself and ask another group.  In this fashion I attempted to record one group from each of the “attentive” and “disinterested” categories during each session that I visited. Before commencing observation of a student group, I re-emphasized that the purpose of the research was merely to observe students’ normal activities and practices and would have no impact on their grades.  In an attempt to minimize the disturbance to students’ normal interactions during the recitation session, I kept my body language separate from the group’s, and endeavored   119 not to engage students with my gaze or body language.  I occasionally glanced at students directly in order to make note of their body language and gaze behavior. A minimum of 4 student groups from each of recitations 7, 8, and 9 were recorded: half only with a simple audio recording device, and half audio- recorded while I sat with the students taking field notes. A total of 14 small- group recitation sessions were recorded.  Each session was approximately an hour long and was transcribed for analysis.  An individual session is referred to as an episode in the remainder of the dissertation.  A subsection of a particular episode is referred to as a snippet. 6.3.2 Coding Problem-Solving Prompts As described in section 2.4.4, students are required to use a prescribed problem- solving strategy in the Physics 100 recitations. This strategy lays out a series of 6 steps intended to lead students through an expert-like problem-solving process. This strategy is taught and used in lectures, and is also written on problem solving worksheets that are given out to each problem-solving group in the recitation. The worksheets have the problem written on the first page, and then each step is listed along with some notes elaborating what is expected for that problem-solving step. Below each step is an appropriate amount of whitespace for the students to write their work for that step. The steps and explanatory notes are quite general, and are the same from week to week. Much of the research in constraining novices to use expert like problem-solving strategies focuses on requiring students to engage in “higher-level” thinking which is intended to enrich their procedural and mathematical treatment of physics problems by encouraging them to consider relevant conceptual and qualitative knowledge such as physics concepts, the narrative of the problem, and their common sense.  In developing this problem-solving method, the Physics 100 instructors and I expected that the descriptive prompts for steps 1, 2, 3, and 6 would encourage students to make use of their conceptual and qualitative information.   120  To enable me to examine the correlations between the prompts and students’ use of conceptual or real-world knowledge, I coded the transcripts to determine when students reached each step of the prescribed problem-solving method.  Often, when reaching a new step students would either read that prompt out loud or make a direct comment about “what we have to do next”. In circumstances where the timing of reaching a particular step could not be determined,  the discourse in that prompt was excluded from my analysis. Coding of the prompts was verified by two researchers who coded three different transcripts independently with 100% inter-rater reliability. The remainder of the prompts were coded by a single researcher. Note that I do not code what kind of problem-solving activity the students are engaged in, but simply note the time when they reach the printed problem-solving prompts on their worksheet.  Because the goal of this study is to examine how students react to the pedagogical structure of the prescribed problem-solving strategy, I am interested in how they respond to these prompts. 6.3.3 Validation of Audio-Only Coding of Epistemological Framing To code epistemological framing for an interacting group of students throughout the recitation episode I used a modified version of a coding scheme developed by Scherr and Hammer [64].  Rather than use video analysis, as was used by Scherr and Hammer, I reasoned that it would be preferable to conduct framing analysis on audio data for my research.  Audio data is much cheaper and easier to gather and engenders fewer ethics concerns regarding the participants’ anonymity.   In order to investigate whether behavior-based epistemological framing coding could be conducted without the aid of video, I collaborated with Dr. Scherr to compare the results of two different researchers coding the same data but using different media modalities.  One researcher coded audio and video together, and the second researcher coded audio only.  In addition to validating the methodology used for my thesis, we reasoned that a proof of concept for audio analysis of framing might open up this research methodology to a wider variety of researchers.   121 6.3.3.1 Data  In order to investigate the correlation between audio-only and video coding, Dr. Rachel Scherr and I coded four different episodes. Three of the episodes were videos upon which the original Scherr and Hammer paper had been based. The fourth episode was an audio-only episode from the Physics 100 recitations.  These episodes are summarized in Table 19 below. Episode Rachel Coded Sandy Coded Purpose 1 Audio Audio To educate Rachel about audio-only coding, so we could discuss the relative benefits of each medium 2 Video Video To train Sandy in Rachel’s coding scheme; to replicate original result 3 Video Audio To compare relative merits and reliability of audio-only vs. video 4 Video Audio Table 19:  Summary of audio and video episodes coded for validation of audio-only coding of epistemological framing. 6.3.3.2 Methodology All four episodes were coded using the Scherr and Hammer coding scheme, which described in section 6.2.1.1 above. Both researchers coded the audio episode in order to investigate the limitations of audio–only coding. One of the video episodes was coded by both researchers using the full video data in order to conduct a replication study of the original Scherr and Hammer result and to ensure that I was trained to use this coding scheme in the same fashion as the researchers that coded for the original paper.   The two remaining video episodes were coded by me using audio only. This coding was compared to the audio and video coding that was used in the original Scherr and Hammer study. To code epistemological framing using audio signals we made use of the audible behavior described in the original Scherr and Hammer paper, as well as identifying some additional audio cues that could indicate student behavior. Below I will review the audible signals that we associated with each frame.   122 BLUE:  WORKSHEET FRAME In this frame students’ primary concern is getting things down on the worksheet.  In many cases this means the students do not speak, and the audio is silent or faint scribbling of pens or flipping of pages can be heard.  It was assumed that any long period of silence with on-task discussion bracketing it indicated the students were in the Blue frame. In this frame students may also “check in” with each other in order to verify something that they are about to write down.  These verbalizations are usually fairly short and consist of one or two brief question-and-answer exchanges.  The vocal pitch is not dynamic, and tends towards monotone or simple rising and falling tones to indicate questions and answers.  These check-ins are typically bracketed by silence. The silences between exchanges is a key indicator that the students’ attention is not fully focused on having a discussion with each other.  In a recitation context, if the bracketing discussion is on-task, we assume that in the intervening silences the students are still attending to the problem worksheet. Another vocal signal that indicates the worksheet frame is speaking slowly with elongated vowel sounds and/or long pauses between words.  This indicates that the speaker is writing, and is speaking aloud at the same tempo as their writing.  This behavior of writing may also be accompanied by muttering in a barely audible way. GREEN:  DISCUSSION FRAME In the Green frame students speak clearly.  Their voices are loud enough to be clearly heard by their peers, and enunciation is crisp enough to be understood. The rhythm of speech is more rapid.  Statements may be longer and more elaborated than in the Blue frame, but responses come very quickly.  There are typically very few noticeable pauses between one student’s statement and   123 another’s response, and students sometimes interrupt each other or finish each others’ sentences. The pitch of the voice is more dynamic.  As students argue and reason with each other their vocal pitch may change in ways more complex than the simple question and answer patterns observed in the Blue frame. One of the more important ways to distinguish between Blue frame (Worksheet) and Green frame (Discussion) is the speed with which students respond to each other in conversation.  In the Green frame students’ responses are very rapid, sometimes even interrupting each other with a new idea or finishing each others’ sentences. RED:  TA FRAME In this frame students’ attention is on the TA.  From the audio data it is impossible to tell where the students are looking, so we assume that they are in the Red frame whenever the TA is present at the table or whenever the TA is speaking to the entire class. YELLOW:  JOKING FRAME To code the audio for this frame we look for students using a joking or sarcastic tone of voice.  Students in this frame may exaggerate the normal variation in pitch.   A joking remark may be followed up with several more. Laughter is also an indicator of this category. 6.3.3.3 Results After both researchers had coded each episode their codes were compared. As in Scherr and Hammer’s original study, frame transitions that were within 5 seconds of each other were considered to be the same within a reasonable margin of error, and were therefore not counted towards the total amount of error. Any other mismatches were counted as errors, and the final inter-rater   124 reliability was calculated as IRR = 1- (errors) / (total duration).   The results of this comparison are summarized in Table 20 below. Episode Coding Type Duration (sec) Errors (sec) IRR 1 A/A 1515 155 90% 2 V/V 1870 190 90% 3 A/V 1425 235 84% 4 A/V 3010 525 83% Table 20:  Inter-rater reliability for study of audio-only vs. video coding of epistemological framing.  Frame transitions were coded to within 5 second accuracy.  The inter-rater reliability was calculated as IRR = 1- (errors) / (total duration coded).  As shown in Table 20, all episodes were coded with greater than 80% inter-rater reliability.  This suggests that much of the framing information that is coded in body language is also present in gestural characteristics of the voice. More attention must be paid to the specific rhythm and characteristics of the voice, but by attending to pitch, rhythm, speed of reply, turn–taking in conversation, and other aspects of prosody students’ expectations about communication and collaboration are revealed. Although the sample sizes are obviously very small, it is interesting to note that the highest inter-rater reliability occurs when both coders are using the same media type. In order to investigate the cause of the errors, the 17% of time where the coders disagreed on episode 4 was discussed in order to determine the reason for this agreement. A total of seven main categories accounted for 84% of the disagreement, and these seven categories could be broken up into 2 main themes: issues with the audio only methodology, and refinements or corrections to the original coding. The breakdown of these errors is depicted in Figure 12 below.   125 Figure 12: Summary of reasons for inter–coder error when coding epistemological framing with audio only vs. audio plus video. 6.3.3.4 Limitations of Audio-Only Coding for Epistemological Framing As shown in Figure 12, 42% of the errors were due to issues with the audio methodology. These errors highlighted weaknesses of using audio only in order to perform a behavioral coding. For example 13% of the errors were due to circumstances where the students gave unambiguous body language cues but their audio cues were ambiguous. An example of this was when one student was speaking very clearly about his ideas without interruption or elaboration from his peers. The lack of audible interaction does not give any information about the actions of the other group members, but their silence suggested they were paying attention to their peer’s speech. However the video showed clearly that the other group members were not paying attention and were instead focusing on their worksheets. 83% Agreement 17% Error clear behavior, unclear audio 13% speech / behavior mismatch 12% discussion during the TA frame 13% difference in coding bid vs takeup of frame 4% mixed frame 16% original coding based on content of speech 7% change in original coding 19%   126 Another type of issue with coding using audio only arises when students’ speech and behavior explicitly give different signals. An example of this is when students are speaking clearly to each other while keeping her body language oriented towards the worksheets. Their clear speech indicates to the audio coder that the students are engaged in discussion with each other, but their attention to the worksheet and static body language indicates to the video coder that they are focused on the worksheet. The third category of problems with using audio only for coding of epistemological framing is that it is impossible to distinguish whether students are paying attention to the TA when the TA is physically present. There were several instances where students became so engaged in a conversation with each other while the TA was present that the video researcher coded their conversation as Discussion rather than as Paying Attention To The TA. However there are no discernible audio cues for when this occurs and so the audio–only coder has no choice but to code the TA frame for the entire duration of the TA’s presence at the table. Finally, there is an important difference between the two coders’ ability to detect a group’s transition into a new frame. When one has access to the video record of the student body language, one can easily tell when the students are paying attention to each other. This makes it easy to tell when a particular student's bid for frame change has been taken up by his or her colleagues: their attention to the new idea will be apparent. However when one is coding audio only, one must wait for some verbal evidence that a student’s peers have begun paying attention to his or her idea. The necessity of waiting for a verbal cue to indicate a group’s subscription to a new frame leads to small differences in when frame transitions are coded by the audio–only coder versus the audio plus video coder. 6.3.3.5 Comments on Scherr and Hammer's Original Coding The comparison of the two coding methods also revealed some issues with the original coding. Firstly, upon comparison and discussion of some of the   127 coding errors Dr. Scherr agreed that some of the video codes that had been assigned during her original research did not match with the coding scheme as described in the paper.  As such, when I coded the audio according to the published coding scheme, the code that I assigned was different than the code assigned during her original research. Secondly, despite the original paper’s claim that the coding was based on students’ behavior alone, it was revealed that some of the original codes were made based on the content of student speech. This coding based on content added additional insight into how the students were framing the current situation, but violated Scherr and Hammer’s claim that their methodology used only the behavioral characteristics of speech and body language in order to assign an epistemological frame code. In these cases the consideration of the content of students’ speech led to a different code being applied, which was later revealed as a difference between the two coders’ results. Finally, a re-examination of the disagreements between the two coders showed one instance where the students' behavior did not fit cleanly into one of Scherr and Hammer’s four categories. The students engaged in a protracted, thoughtful, dynamic discussion [Green frame] that was completely concerned with and directed towards the worksheet [Blue frame]. This behavior was clearly a mixture of the two frames listed by Scherr and Hammer. 6.3.3.6 Summary An investigation of the inter-rater reliability between one rater who coded epistemological framing using audio only and another rater who coded using both audio and video showed that these raters coded the same epistemological frame more than 80% of the time. While there are some weaknesses of using audio only for this purpose, this high reliability leads me to conclude that audio gives sufficient information to code students’ epistemological framing.   128 6.3.4 Development of Epistemological Framing Coding Scheme In addition to coding via audio-only, I also modified the Scherr and Hammer coding scheme in two ways.  This coding scheme was adapted for suitability to my particular context and students, and was refined by using examination of the implicit goal of students’ discussion to identify two different epistemological frames that occur during discussion. In the following sections I describe the adaptation of the Scherr and Hammer scheme and the new frames that were particular to this context.  Then I summarize the final scheme and give some representative coding examples, describe the inter- rater reliability study used to refine the scheme, and discuss the limitations of this coding methodology. 6.3.4.1 Adaptation to Cohort For their study of behavior and epistemological framing in group problem solving, Scherr and Hammer pre-selected the groups to study based on which ones talked to each other frequently and were consistently on task.  By their own description, this represents a best-case scenario. While this methodology seems appropriate for an exploratory study aimed at identifying different categories of students’ frames, my aim is slightly different. I want to make arguments about correlations between frames, features of the recitation structure, and students' thinking. Therefore I have endeavored to gather data from a broader variety of student groups. Indeed, in order to avoid analyzing only the "best case," I deliberately solicited the participation of students that seemed disengaged in the recitation format in addition to students that volunteered and expressed positive opinions of the recitation and the course.  As well, some of the groups studied did not interact with each other in an ideal interactive and collaborative fashion.  The hope is that this special attention to recruiting a broad variety of student groups will yield results that are more representative of the student population as a whole.   129 6.3.4.2 Conceptual and Procedural Discussion Another important extension of the Scherr and Hammer coding scheme was the splitting of their Discussion frame into two subcategories. These two categories arose from going beyond behavior–only coding to consider the content of the students' speech. Specifically, when considering the transcripts I noticed that there were two broad goals that motivated students in their discussions:  understanding of the current ideas or figuring out how to make progress towards a solution.   As discussed below, these different goals are evidenced by students’ speech and have an influence on their behavior in the recitation as they try to achieve their goal.  Because these two goals represent an important distinction in the students’ sense of the essential nature of their current activity and because these different goals have different implications for how students negotiate and develop knowledge, I argue that they distinguish two different epistemological frames.  The goal of understanding of the current ideas corresponds with a frame I have labeled Conceptual Discussion (CD), and the goal of figuring out how to make progress towards a solution corresponds with a frame labeled Procedural Discussion (PD). Unlike the changes in behavioral clusters marked by Scherr and Hammer, the switch between Conceptual Discussion and Procedural Discussion cannot necessarily be distinguished by changes in tone, rhythm, or register.  In the tradition of Tannen, I make my arguments about the existence of these frames based on the substance of speech as well as linguistic markers associated with speech acts [63]. To flesh out our description of these frames, I will show snippets of transcript where students’ implicit goals are clear.  Then I will describe the markers of the frames used in my coding scheme and compare and contrast these frames to each other and to Scherr and Hammer’s frame.  Throughout these descriptions I will be sharing my interpretations of students’ goals and meta-messages, interpretations that I believe rest on a shared cultural language of framing communication which I expect that I share with many of my subjects   130 and readers.  I hope that my readers will agree with my analysis of these quotes, supporting my interpretations. CONCEPTUAL DISCUSSION Some student discussions are clearly oriented towards interpreting the meaning of the problem, building an understanding of the physics, or developing a coherent narrative or visualization of the situation under consideration. This frame is where we see students striving to express, understand, and synthesize new ideas. This frame has the implicit goal of “figuring out what is going on,” and is called the Conceptual Discussion (CD) frame. The following example illustrates this frame.  A group of students has just read a recitation problem which centers around the force in ropes that are restraining a fridge in the back of a pickup truck.  (See page 316 in Appendix A for the full problem)  The transcript begins with Student 2 asking a question after a long period of silence, during which the group has read the recitation problem. Line Speaker Quote 1 Mary But if there's 2 of the straps does that mean you can double the force? 2 Natasha Are there 2 ropes or 1 rope? 3 Leah There's 2 right? 4 Natasha 2 ropes? 5 Mary I don't know 6 Natasha Or it's the same rope and it's (mumbling) 7 Mary If it's like each rope takes half that 8 Leah Oh like that, yeah.  I think that would make sense, because it said that it was tied at the back, so that doesn't make a lot of sense 9 Mary No Table 21:  Example of Conceptual Discussion.  The students’ focus is on figuring out what is going on in the situation and the implications for the force in the rope.   131 In this episode, the students are trying to visualize the configuration of ropes that hold the fridge in the truck. The episode begins with Mary asking about the number of straps as well as the implications for the force on the fridge. Her question on line 1 clearly targets the conceptual relationship between the number of ropes and the force in each rope. In the subsequent dialogue the students discuss different possibilities for the configuration of ropes.  Leah’s comment on line 8 demonstrates that she is evaluating these possibilities in terms of “what makes sense.”  This implies that Leah’s implicit goal in this conversation is constructing a picture of the truck, ropes, and fridge that meets his expectations about sensible arrangements of trucks, ropes, and fridges. It is interesting that, on the surface, that Mary and Leah are focused on different aspects of the problem.  Mary is focused on how the two ropes would double the force, whereas Leah is trying to work out a sensible visualization of the problem.  However, both are engaged in trying to interpret the meaning of the problem by coordinating it with their conceptual understandings and expectations about the world, and in that sense share an epistemological frame. This shared focus on determining “what makes sense” is indicated by Mary's response on line 9, which is delivered immediately after Leah’s statement and with a tone of affirmation.  By this immediate response Mary indicates her agreement with Leah and also implies that she believes that discussing “what makes sense” is the appropriate activity in this moment.  This immediate, implicit agreement implies that Mary and Leah are sharing the same frame. Not every instance of students interpreting a problem statement can be considered Conceptual Discussion.  Many such instances involve students working to understand what the professor is asking without considering their own judgments of what kinds of physical systems and configurations would be meaningfully coherent.  However, the students in this transcript are engaged in developing a coherent visualization and understanding the force in the ropes in ways that foreground their own conceptual knowledge and sense of real-world   132 plausibility.  While these students will turn their attention to the requirements of the problem shortly, in this moment they are more concerned with their own requirements for a coherent picture that makes sense to them.  This shared focus on trying to develop a coherent and meaningful visualization is an excellent example of “figuring out what is happening,” and therefore a good example of the Conceptual Discussion frame. PROCEDURAL DISCUSSION At other times students’ discussions are clearly oriented towards figuring out what to do in order to make progress.  In this case, “progress” is defined in terms of the external or formal requirements such as the instructions on the worksheet, perceived expectations of the professor, or completion of mathematical calculations.  Students may discuss and reason with each other in order to choose a course of action, but their warrants are drawn from either a sense of what they “should do” next (referencing the external authority of the professor or worksheet) or from a sense of mathematical convenience rather than reasoning based on what makes sense to them.  This frame is identified by the implicit goal of “figuring out what to do,” and is therefore called the Procedural Discussion (PD) frame. The notion of discussion that is not oriented towards physics concepts marks a significant distinction from the Scherr and Hammer scheme, which claims that when engaged in discussion students are always discussing each other's ideas. Based on my observations of the content of student speech in the Physics 100 recitations, I claim that students can be engaged with each other in discussion of how to proceed in the problem. My observations have shown that these discussions exhibit authentic responsiveness to each others’ ideas and dynamic vocal register (hallmarks of the Scherr and Hammer Discussion frame) but are focused on procedural rather than conceptual matters. The Procedural Discussion frame also borrows some of the characteristics of the Scherr and Hammer Worksheet frame.  Students in the Procedural   133 Discussion frame are often concerned with how to satisfy the professor's requirements or to get the marks on the recitation worksheet, characteristics which Scherr and Hammer associate with their Worksheet frame.  However, the characteristics of students’ speech in the Procedural Discussion frame are dynamic and interactive, characteristics associated with Scherr and Hammer’s Discussion frame, which they claim indicates that the students are framing their activity as figuring something out. Based on their dynamic speech behavior, I contend that the Procedural Discussion frame is essentially understood by the students as a Discussion.  In this case, the thing that they are figuring out is a strategy, a plan for their calculation, the requirements of the professor or the worksheet, or anything else that will help them proceed towards the goal of completing worksheet. The similarities between my Procedural Discussion frame and the Scherr and Hammer Worksheet frame raise the question of whether Procedural Discussion ought to simply be coded in the Worksheet category.  Based on in- class observations that students are looking at each other during Procedural Discussion frames, I would argue that this frame is distinct from the Worksheet frame, where students’ attention is directed to the worksheet itself.  As well, the dynamic tone of voice and rapid, responsive conversation are much more like the behaviors associated with discussion than the low register, monotone, clipped, or broken speech associated with the Worksheet frame.  In my scheme, the Worksheet frame is reserved for the doing of the worksheet: the immediate business of executing the group’s plan, conducting calculations, and/or writing down the results. The transcript below shows an example of a Procedural Discussion frame. This transcript is drawn from a different group working on the same problem as above.   As above, this transcript begins just as students have finished reading the recitation problem which centers around the force in ropes that are restraining a fridge in the back of a pickup truck.  (See page 316 in Appendix A for the full problem)   134 Line  Speaker Quote 1 Igor So the second paragraph says you don't want the hitch to come loose and smash, so basically if it's there, and you have a collision you'd be like, whooop 2 Winifred ok 3 Unknown (Background noise, pages turning) 4 Igor Yep, so we're calculating the deceleration 5 Denise Umm, well we're calculating the force exerted on the fridge by deceleration 6 Igor Yep, and if it will exceed the maximum like 750N? You know, pretty, straightforward 7 Winifred So I guess our goal is to determine whether the deceleration force is greater than or less than - the deceleration force exerted on the fridge - is less than or equal to 750N? 8 Igor Yeah 9 Winifred Is that right? 10 Denise I think so (pauses) 11 Winifred I'm not exactly sure how we would calculate that 12 Igor Well we don't really need to know the truck's mass do we? 13 Denise (interrupting) For the force? 14 Igor Or no, we do, for the acceleration 15 Winifred No, I think all we need to know is, um, I think we can use like v_1, v_final, acceleration, distance, cause we know the distance and then we know v_initial and v_final, we could find the deceleration, and then we can find the deceleration in terms of g, right?  (pause) um, which, multiplied by the kilograms of the fridge is going to give the number of Newtons?  Is that right? Table 22: Example of Procedural Discussion.  The students’ focus is on figuring out what they should be calculating. The transcript begins with Igor’s interpretation of the goal of the problem, which is to ensure that a fridge is tied down adequately so that it wouldn’t come loose in an accident.  This interpretation is quite qualitative and conceptual, but does not elicit much response from the group except for a simple “ok” from Winifred.  In the next statement on line 4 Igor announces “what we’re doing,” an implicit way of soliciting his group’s consensus for his proposed plan.  His conception of “what we’re doing” is completely concerned   135 with the result of a calculation, an approach which is reinforced by the contributions of Denise on line 5 who discusses what “we’re calculating,” and by Winifred on line 7 who describes that their goal is to determine the mathematical magnitude of a certain force.  Throughout the remainder of this transcript the students’ conversation is limited to discussion of what should be calculated and how it ought to be calculated. In this segment the students respond to each others’ ideas, follow a thread of conversation, and communicate with each other in clear, dynamic voices that indicate they expect to be heard by their peers.  These behaviors display the students’ framing as having a discussion, but the discussion is centered entirely around the perceived requirements of the worksheet and calculation. In order to address those requirements, the students access their knowledge about calculations, a very different set of resources from the students in the same situation above who discuss physical configurations, objects, and relationships. This illustrates how the implicit goal of figuring out what to do describes a distinct frame that has different implications for what kinds of knowledge and actions students are likely to see as fruitful.  Therefore, defining the distinct Procedural Discussion frame is both necessary and justified. The addition of a distinctive Procedural Discussion frame raises the question of why Scherr and Hammer did not observe a similar cluster of behaviors in their study.  I hypothesize that this type of engaged-discussion- about-procedure might be more necessary in our context than in theirs due to the fact that students in Physics 100 have only one worksheet to fill out rather than each student working on their own worksheet.  This places a higher premium on group consensus, which would require students to interact and look at each other more closely even when they are discussing seemingly mundane strategic questions.  As well, our context-rich recitations place a much larger onus on the students to interpret the problem and plan their own solution than the highly-scaffolded tutorials used in the Scherr and Hammer   136 study.  This might serve to further increase the need for students to explicitly develop and discuss their solution strategy, increasing the need for Procedural Discussion.  Finally, the type of tasks given in Scherr and Hammer’s tutorials do not require students to perform extended calculations.  Therefore, protracted discussions of problem-solving strategy may have been simply unnecessary. COMPARISON OF CONCEPTUAL AND PROCEDURAL DISCUSSION Over the course of coding the wide variety of transcripts for this project those two organizing principles of “figuring out what to do” and “figuring out what is going on” have formed the core identity of the Procedural Discussion and Conceptual Discussion frames.  The students’ focus on one or the other of these goals is evidenced by the warrants they use in arguments, their use of mathematics, the way they attend to the description and instructions written on the recitation sheet, as well as direct references to procedure, planning, and meaning.  Table 23 summarizes some of the characteristics of these two frames.   137  PD:  Procedural Discussion. CD:  Conceptual Discussion. Primary Goal GOAL is to make progress towards the answer of the worksheet. Interpreting what the instructor / worksheet is asking for.  Satisfying the different steps of the process.  Getting an answer on the page GOAL is to interpret / understand what’s going on.  Students want to satisfy their own sense of what is sensible / reasonable. Focused on Focused on the worksheet, satisfying worksheet structure Focused on understanding the physics of what’s happening Example Activities and Quotes Discussion of strategy – “what do we do next?”, “we can do X or Y” Coordination of representation from narrative -> mathematical  “Vf is zero because it stops”  Discussion of “what we’re supposed to do”, “should”, “what do we need to do” Seeking to understand meaning of physics concepts "How can it be both constant acceleration and velocity both?"  “What we’re calculating” Reference to students’ own intuition or commonsense  "that would make a lot of sense" Review of Problem Paraphrasing and / or interpreting given problem in order to identify knowns & unknowns Conceptual interpretation of the given problem / situation  Interpretation or  visualization to develop a  narrative that makes sense to the students. Discussion of Assumptions Discussion of assumptions framed as a list of things to satisfy worksheet condition Discussion of assumptions framed as what makes sense. Interpretation of Mathematics Interpretation of consistency of mathematics Interpretation of MEANING of mathematics, or MEANING / concept / Narrative of what’s happening in the question  Conceptual interpretation of mathematical structure.  “This part is the heat in, and this part is the heat out” Evaluation of Results Evaluation of results that is brief, shallow, parrots prior work / authority. Evaluation of result by “looking at the big picture”  Evaluation of results in terms of algebraic consistency Evaluation of results in terms of qualitative / conceptual understanding of the result. Table 23:  Characteristics of Procedural Discussion and Conceptual Discussion frames. 6.3.4.3 Differences in Worksheet Frame Scherr and Hammer’s Worksheet frame is characterized by an immediate focus on recording things on the worksheet.  When conducting video analysis,   138 this is evidenced clearly by body language oriented towards the worksheet. However in audio we must pay attention to different characteristics in order to infer the object of students’ attention. The first cluster of audible behaviors that indicates a focus on the worksheet is when the students are engaged in writing or calculation. In the best case, one can hear the sound of pencil scratch on paper while the group is writing silently.  Direct observations have confirmed that a period of silence bracketed by on-task discussion means that somebody is conducting a calculation or writing something down.  While writing or calculating students may mutter to themselves or recite what they are writing.  Students that are reciting as they write will have a noticeably slowed pace of speech with pauses in between each word that regulate the speech to the same pace as writing. In normal conversation these pauses would be unnecessary and perhaps annoying but when recording what one presumes is the consensus of a collaborative group, this recitation serves as a last chance to ensure the group's consensus on what is to be written. In my analysis, I found a second main category of behavior that corresponds to a focus on the worksheet.  The second cluster of behavior that occurs in the worksheet frame is when one student is writing and one or more students are directing them what to write. A student will sometimes feel it is necessary to closely monitor and/or directly control what is being written on the page even though they are not the person with the pen. In these circumstances the audio recording will show a steady stream of direct instructions to the person writing.  Typical behaviors that indicate this is occurring are: the person directing speaking very slowly, pronouncing syllables individually, pronouncing punctuation marks, repeating themselves, or offering explicit imperative directions such as “write X squared”.  While these speech acts may have very clear diction and are clearly intended as communication, their imperative nature demonstrates that they are instructions to another person and are intended to directly influence what is being written.  This focus on the writing   139 on the page indicates that the students engaged in this behavioral cluster are also focused on completing the worksheet.  Therefore, this cluster is subsumed into our definition of the Worksheet frame. This behavioral cluster is absent from the Scherr and Hammer data, likely due to the fact that in their study each student has their own copy of the worksheet. 6.3.4.4 Comparison of Procedural Discussion and Worksheet Frame One might argue that the Procedural Discussion and Worksheet frame, while behaviorally different, are both concerned with the same epistemological activity of generating answers to the worksheet questions.  There are certainly many similarities in this regard.  In both frames, the students’ conceptual understandings take a backseat to their focus on understanding and meeting the perceived requirements of the problem.  However, there is an important difference between these two frames that supports my practice of treating them separately.  In the Procedural Discussion frame, the students are engaged with each other in the process of figuring out what to do.  That is, they are developing a strategy for meeting their perception of the goals of the worksheet. Conversely, in the Worksheet frame, the students are executing their strategy. The relative paucity of conversation reveals their belief-in-the-moment that no more discussion or negotiation is required in order to determine what to do next; all that is required is the execution of the previously-agreed-upon plan. It is interesting to see these two frames interleave with each other during the problem-solving process.  Students will develop a plan, begin executing it, and then when either a result is obtained or an unanticipated barrier is encountered, they will return to Procedural Discussion in order to figure out what to do next.  This pattern recalls Goffman’s notion of “nested frames” [80], in the local “develop a strategy to get the answer” frame as well as the “execute the strategy and record the results” frame may be nested in an overarching “complete the worksheet” frame.  Procedural Discussion and Worksheet frames may both be oriented towards the overall goal of completing the worksheet, but   140 their different orientation towards developing vs. executing the solution strategy come with different epistemological commitments, and are therefore appropriately classified as different epistemological frames. 6.3.4.5 Other New Frames Due to particular features of our instructional context and research goals, two other distinct frames were identified. The first of the special frames deals with behaviors that do not fit into any of our other frames and that we felt were important to our goal of exploring how students make connections between their everyday knowledge and the recitation problem. In this frame, students comment on and criticize the ostensible connections between the problem and the everyday world. This may take the form of praise or complaints about the perceived realism of the problem situation itself. Because these speech acts are not focused on completing the problem or understanding the physics concepts that are promoted by the problem, they do not fit into either the Procedural Discussion or Conceptual Discussion frames. In order to reflect the students' understanding of these situations, and also to mark what we believe are important moments of connection between the recitation and the real world, these conversations are marked with the “Meta-Comment” frame. The second special frame arises as a result of our use of the prescribed group roles recommended by Heller et. al for use with their context-rich problems [1].  Students are assigned a different role each week, and may be required to act as the group's manager, recorder, skeptic, or explainer.  As such, a certain amount of the students' discussion revolves around assignment and duties of various group roles. These discussions are clearly identifiable but do not fit into any of our other frames and so we identify them with the “Group Role Negotiation” frame.   141 6.3.4.6 Summary of Framing Coding Scheme The set of epistemological frames used for this study is listed in the table below.  The full coding guide and rubric that were used for coding epistemological framing is listed in appendix D. Frame Code Description Conceptual Discussion CD Engaged discussion to make sense of the narrative of the problem or meaning of the physics Procedural Discussion PD Engaged discussion to figure out how to proceed or what the professor expects Worksheet W Focus on writing on worksheet or directing others' writing TA focus TA Focus on interacting with the TA Group Negotiation G Focus on interpreting or reinforcing assigned group roles Meta-comments M Focus on discussing (un)realism of provided tutorial problem Other / Off-topic O joking, discussion not having to do with tutorial Table 24:  Summary of epistemological framing coding scheme. Typically, coding is done in two passes over the audio recording. In the first pass, the students' tone, rhythm, and prosody are used to make the main distinctions between Worksheet frame, TA frame, Off-topic frame, and Discussion. Then the coder will make a second pass through the areas marked Discussion paying close attention to the content of students' speech. The details of exactly what students are saying and implying are used to distinguish between Conceptual Discussion, Procedural Discussion, Group role negotiation, or Meta-comments. 6.3.4.7 Example of Framing Coding The following example demonstrates the Procedural Discussion, Conceptual Discussion, Worksheet, and Off-Topic frames as well as several frame transitions.  These frame transitions help to illustrate the frames by contrasting them against each other.  In this transcript a group of students are   142 working on a problem that concerns calculating the force on a passenger during normal braking and also during a crash.  Prior to the beginning of this transcript the students have been instructed to “Define Assumptions and Relationships,” one of the standard steps in the prescribed problems solving method used in the Physics 100 recitation questions.  (See section 2.4.4 for more details on this problem-solving method.)  The students have already been discussing their assumptions for several minutes.  The transcript begins as S4 brings the group’s attention back to this task after a short period of off-topic discussion.   143 Line Speaker Quote RWC Frame 1 Michael Ok, what else? Well, we assumed that, um (pause) no airbag. 1 W 2 Gordon All right 3 Isaac Isaac:  Is he the one that’s driving? No, his friend’s driving; he might just be in the back seat. Oh yeah.  No, wait 4 Michael Assume he’s in the front seat too, that’s a good point because he would hit the seat and that would disrupt our calculations. 1 CD 5 Isaac no, you could still hit your seatbelt before hitting the seat. 1 6 Kirk you could still hit the … dashboard 1 7 Michael That’s true 8 Kirk just assume that 9 Isaac part of why we have seatbelts is so you don’t fly into the seat. 1 10 Kirk Assume the seatbelt stops him before he hits something. 1 11 Michael That would suck. 12 Isaac That would suck so hard. 13  I mean you’re either sitting behind a seat and you fly into the seat, or you’re not sitting behind a seat and you fly out the window 14 Kirk I mean I would way rather fly into the seat than fly out of the window. 15 Michael Have you ever been hit in the face by an airbag?  O 16 Kirk I had a guy - a guy in my high school went through the windshield, and he had the most like serious type of concussion.  He had, through his  17 Isaac Sometimes it’s actually better to be outside of the car than inside. 18 Michael I took an airbag in the face when we were doing like 30, and it sucked. It was like “doing” 19 Isaac "boom" 20 Michael Yeah you wake up like 2 minutes later and it’s like, "Um?" And you're bleeding all down your face and shit. 21 Isaac And you got like burns on your faces too right? 22 Michael It’s not very fun. We weren’t going very fast. If the airbags hadn’t fired we wouldn’t have gotten hurt at all. 23 Isaac (laugh) Table 25:  Transcript to illustrate Procedural Discussion, Conceptual Discussion, and Off- Topic epistemological frames.   144 Line Speaker Quote RWC Frame 24 Michael Because the airbags fired, both of us were bleeding out of our noses and all 25 Kirk They get - they fire out at like insane speed 26 Isaac They, they are fast. 27 Michael 60km or something.  It’s nasty, it’s like getting hit in the face with a brick wall. It sucked. 28 Kirk We assume the mass is 100k, we assume he’s wearing a seatbelt.  PD 29 Michael This is for the first situation right? Or this is for both situations? 30 Isaac Yeah they work for both. 31 Kirk There's no airbag, the seatbelt stops him before he hits something. 32 Michael Ok, perfect.  What other assumptions have we got? 33 Kirk Anything else? 34 Isaac He is a man. We’ve been assuming that the entire time 35 Gordon (laugh) well 36 Michael Car stops in either 37 Kirk Well we’re all guys 38 Michael I don’t know if those are assumptions.  The car stops in either 1 or 20 meters. Is that an assumption? It’s kind of given to us isn’t it? 39 Gordon yeah I think it’s given 40 Isaac It’s one of the like basics for solving the question 41 Michael It just says that 20 meters per hour is the stopping distance, so let’s just assume that the manual is correct for the car and that it actually stops  42 Isaac Assume that the stopping distance is actually 20 meters 43 Michael Yeah I think that’s important 44 Isaac Also assume that an emergency stop would be exactly at 1 meter. 45 Michael That’s good, and I think we’re pretty good. 46 Kirk We gotta draw the model and then we gotta solve. 47 Isaac Draw some pretty pictures, draw some free form diagrams. Table 25:  Transcript to illustrate Procedural Discussion, Conceptual Discussion, and Off- Topic epistemological frames    145 The above transcript includes four segments:  students working in a Worksheet frame, a Conceptual Discussion frame, an Off Topic frame, and a Procedural Discussion frame.  Throughout this transcript the students are engaged in discussion with each other, but the implicit goal of their conversation shifts a couple of times. The transcript begins with the interrogative “what else?” and the subsequent few lines are conducted in a clipped, matter-of-fact tone of voice.   This question and the tone of the exchange imply the students are framing their current activity as filling out a list of requirements. This activity of filling out requirements is a common activity during these recitations, and is characteristic of the Worksheet frame. When the students are framing their activity in this way their discussion revolves around interpreting the requirements and searching for the right answers that will satisfy the requirements.  As is evident in lines 1 and 2, there is little discussion of the merits of listed items. The implicit goal of this conversation begins the change in line 4. Michael proposes another assumption to be added to the former list but then expands this assumption by interpreting its consequences for the calculation:  “that’s a good point because he would hit the seat and that would disrupt our calculations.”  This suggests consideration of the implications of the assumptions in terms of the actual narrative of the situation.  Isaac responds on line 5 by making a qualitative argument about the sequence of events in the crash.  The subsequent discussion between Kirk, Michael, and Isaac makes it clear that the focus of discussion has shifted from listing assumptions to conceptually exploring the interactions of a seatbelt and a seat during a car crash. These discussions inform the students' assumptions in the problem and their subsequent calculations.  This example of reasoning from conceptual or narrative representations of a problem to make claims in mathematical or logical representations is a key characteristic of the Conceptual Discussion frame.  Instead of speaking strictly in terms of the elements and rules of formal   146 mathematical reasoning, the students are coordinating their understanding of the story of the problem with the mathematics. At line 15 the discussion of car crashes prompts Kirk to relate a personal anecdote about a car crash. While there is physics knowledge brought up during the discussion, it is clearly not focused on the understanding specific content of the problem or making progress towards the solution, and as such it is coded as Off Topic. The students resume their focus on filling in the assumptions list on line 28, where Kirk recounts their earlier assumptions.  Michael immediately responds with a question relevant to the problem, indicating that he has abandoned the previous line of (off-topic) conversation and is now re-engaged in solving the problem.  The subsequent discussion shows that the students have returned to considering what other assumptions, if any, are necessary in order to meet the standards of this particular step.  During this step of the transcript there is rapid, responsive commentary on each other’s ideas.  Hence, despite the focus on completing worksheet requirements, this segment is coded as Procedural Discussion. On line 38 Michael raises the concern that he doesn’t “know if these are assumptions.”  This is a telling comment; he almost certainly knows a definition of the word “assumption,” but in this context he is concerned about meeting the formal definition of “assumption” that is required for this step of the worksheet. This concern for meeting the perceived standards of the marker or the worksheet is characteristic of the Procedural Discussion frame. Finally, on lines 46 and 47 students explicitly discuss their next steps.  This strategic discussion matches the goal of “figuring out what to do” quite closely, and is also coded as Procedural Discussion. 6.3.4.8 Framing Inter-Rater Reliability Testing To verify the reliability of using this frame coding scheme, an Inter-Rater Reliability (IRR) study was conducted. Two researchers independently coded   147 the same transcripts, coding each segment of the transcript as one of the frames in this scheme. The inter-rater reliability was calculated based on the percentage of frames that both raters coded similarly. One of the key assumptions in our coding methodology is that each member of the group is in the same frame at the same time. However, examination of the dynamic of the bid and take-up of frames shows that frame transitions take some time and during frame transitions not all of the group members are in the same frame. When coding from audio-only data we must often wait for one of the group members to provide an explicit, verbal take-up of a new frame before we have clear evidence that the group has indeed changed frames.  As such, it is hard to be sure of whether a frame change has occurred until at least two verbal statements have been made in the new frame: a bid to enter a new frame and a response indicating take-up of the frame.  By contrast, one can make inferences about a group’s take-up of a new frame from video data by examining the group members’ body language, which provides much clearer and quicker feedback.  This means that our coding scheme has an inherent uncertainty in the timing of a frame change of approximately two statements. This uncertainty in the location of the frame change is a significant challenge for attaining high inter-rater reliability of coding. While we attempt to code the beginning of a clear and unambiguous new frame right when it begins, for a more ambiguous bid we often code in the first location where we have clear evidence that a second student has taken up the new frame. This difference in when a new frame may be coded leads to many situations in which two coders will choose a different beginning point for new frames. Rather than treat these small differences as significant, I instead admit the fact that frame changes are an interactional phenomena, and as such do not occur in an instant. Rather they are the result of communication between group members and the process of proposing a new frame and agreeing to it take time. Therefore, I have chosen to code the inter-rater reliability of frame   148 changes to within 6 seconds. Because the average length of a single statement in the transcript is approximately 3 seconds, a 6 second window of uncertainty on the frame changes corresponds to ± 2 statements which seems appropriate for the interactional nature of this phenomena. With this in mind, the inter-rater reliability is calculated as follows: 1. Each rater codes independently, identifying frame shifts and assigning each statement in the transcript to one of the frames. 2. For each frame transition, if the other coder marked a similar frame transition within 6 seconds, these transitions are considered to be the same and there is no accumulated error. 3. If both coders mark the same frame transition with more than a 6 second difference between the times, the entire difference in time is counted towards the accumulated error. 4. If one coder marks a transition into a new frame and the other coder does not, the entire length of the new frame is counted towards the accumulated error. 5. The inter-rater reliability is calculated as: IRR = 1 - (accumulated error)/(total length of time coded) After coding and comparing a segment, I would discuss with the other coder about our disagreements and make refinements to the coding scheme. This process of coding, comparison, and discussion was repeated until we were able to code 80% or better Inter-Rater Reliability prior to discussion for three consecutive coding sessions without making any further refinements to the tutorial coding scheme, thus demonstrating that the final scheme could be coded consistently. During the process of refining the coding rubric and establishing good IRR we coded five context-rich problems.  The time coded, error rates, and IRR are reported on Table 26 below.  The average IRR for coding of these problems was   149 84%.  All of the disagreements in coding were discussed and we were able to reach a consensus in every case. Episode # Total Time Coded (sec) Total Errors (sec) Total Errors ±6 sec (sec) IRR 1 2096 465 441 79% 2 1201 197 189 84% 3 2402 496 486 80% 4 1193 139 139 88% 5 1800 138 138 92% Total 8692 1451 1393 84% Table 26:  Inter-rater reliability testing for epistemological framing coding scheme.  The IRR is calculated as 1 - (total of errors that are greater than 6 seconds) / (total time coded) Because of the importance of the Conceptual Discussion codes in our analysis, a separate IRR calculation was conducted for only those codes. Using the same 6 second margin of error, the average IRR for Conceptual Discussion for the same five problems listed above was 80%. After establishing this reliability, I used the finished coding scheme to code all of the audio data in the study. 6.3.4.9 Limitations of Coding Methodology After developing this framing coding scheme and using it to code this data set, it is clear that there are several weaknesses of this methodology. One weakness is that decisions about students framing need to be made purely on the basis of audio data. These have been discussed above in the section on comparing audio and video coding and it seems from the studies that we have conducted that this is not a major weakness. The lack of video data requires us to make guesses about students’ attention based on patterns of conversation but without specific information about how students are orienting their body and gaze.  These guesses appear to be mostly correct, but likely rest on shared cultural experience and so may not be generalizable to the circumstance of a   150 researcher making judgments about a context with which they are not intimately familiar. The reliance upon audio data for determining group framing is less of a problem when the group is interacting with each other frequently. It is much easier to infer students attention to each other (or not) when there is speech data available from many members of the group. However it is much more difficult to determine where the group's attention is when one student speaks for an extended period. My work with video showed that in some cases one student might be speaking very clearly while the other members of the group are focused on their papers. Their silence suggests that they are listening and or paying attention, but video reveals that this is not always the case. In this coding scheme, we use a rule of thumb that if a student makes an extended statement (“extended” here being taken to mean more than a few seconds) that silence on the part of your students is taken to indicate attention and implicit assent. While our comparison with the audio-only coding study indicates that this is true most of the time, we know that some of the time students are not paying such close attention to each other. Another issue is that in some groups the dialogue is dominated by one student. In one episode, one student spent the entire session thinking out loud, giving directions to other students, and asking questions which she would then subsequently answer for herself.  Her group mates expressed their support for her decisions and framing very quietly and passively. In order to successfully code the group's attention during this segment, it was necessary to listen to quite a bit of the audio, make a judgment about the normal patterns of conversational turns and dynamics for this group in particular, and then make use of that model of group dynamics to infer the students' attention and subscription to epistemological frames. This raises the other principal weakness of this methodology: the assumption that the entire group is attending to the same thing and framing their   151 engagement in the same way. Again, for a well-functioning group that is collaborating this assumption is likely valid. However there were many instances, even in otherwise well-functioning groups, where there was evidence that one or more members of the group had split off and were engaged in a different activity. Indeed, some groups spent much of their time operating in this split fashion. This splitting of the group's attention is likely enabled and encouraged by the fact that in Physics 100 we only give each group a single recitation worksheet to work on. Many of these split attention episodes occur when one or more students are focusing on conducting calculations and/or recording the group's work. The other students, perhaps feeling that their input is not needed in this moment, will often move on to the next part of the problem or converse about something unrelated. Because this attention splitting behavior was reasonably common, it was necessary to account for it in the coding scheme. Whenever there is clear evidence that different students in the group are attending to different tasks and different frames, a “split frame” is coded. Because of the aforementioned difficulty in evaluating the attention of group members who are silent, this type of split frame is only coded when there is clear audio evidence of two different frames, such as when there are two competing conversations occurring or when one can clearly hear the sound of a student calculating under their breath as they write while the remainder of the group is engaged in conversation. A protracted split in epistemological framing within a group is not mentioned in the Scherr and Hammer study. However I believe this is a consequence of the cohort chosen for their study. Scherr and Hammer's original result was based on video that had been preselected for the most “watchable” groups: groups who displayed a tendency to work together and speak to each other frequently. Groups that displayed a tendency to work independently or were uncooperative with each other were explicitly excluded from their analysis. Because my analysis attempts to span a wider, more representative set   152 of students it is not surprising that I should encounter some groups that are communicating and collaborating with each other at less-than-peak capacity. One final issue with coding from audio only is that, despite using stereo recording equipment, it is not always easy to tell exactly which student is speaking.  Largely, this is not a major problem because our framing coding looks at the group as the unit of analysis so it is not as important to determine precisely which student is speaking. However, because it is occasionally necessary to determine who is speaking for the purposes of evaluating the overall group dynamics or identifying a split frame, this difficulty in determining who is speaking can complicate the coding process. Despite the above-mentioned weaknesses in using audio for coding epistemological framing, we believe that the audio/video comparison study and our own inter-rater reliability study demonstrates that this coding scheme is both reliable and gives substantially the same results as coding using full video. 6.3.5 Coding Real-World Connections A crucial piece of my analysis of students' real-world connections in recitations is operationalizing when students were making a real-world connection.  In order to develop a scheme for coding real-world connections I needed to more clearly define what a “real-world connection” means. One interpretation of the idea of a “real-world connection” in science class is that it is simply a moment where a student makes use of their intuitive understanding of the real world in a science context.  The Physics 100 recitations were explicitly designed to encourage and require this type of connection, with multiple instances where the details of a problem were vague or absent, requiring students to make assumptions and flesh out the situation in a sensible way. I argue that another kind of meaningful real-world connection occurs when a student pictures the immediate scientific idea under discussion as being a part of their real-life or illustrates scientific ideas according to their own expectations of a realistic situation. By working to construct narratives of physics problems that are mutually   153 plausible to both the scientific and the everyday, the students create opportunities for themselves to see science as more connected to the real world. I do not argue that these events necessarily change students perception of the relevance of science the real world, only that they have the potential to do so. To summarize, we operationalize a Real-World Connection as any opportunity for students to integrate their everyday and formal reasoning skills OR to affect their perception of the relevance of physics to the real world.  These moments are characterized by simultaneous activation of physics resources and everyday resources.  (e.g. knowledge, strategies, contextual frames) In the following sections, I will describe the coding scheme used for identifying these two general categories of students’ Real-World Connections in a recitation context.  I will first discuss the goals of the coding scheme.  Then I will describe the development of the scheme and concurrent Inter-Rater Reliability study.  Finally, I will give the details of the coding scheme and illustrate with example transcripts from student dialogue. 6.3.5.1 Development of RWC Coding Scheme To develop this coding scheme, I first started by reviewing student audio and transcripts with the rough definition above in mind and identifying instances where I felt that the students were making meaningful real-world connections. After development of an initial scheme I began working with another physics education researcher in order to refine the scheme to incorporate new perspectives on real-world connections that emerged from the data and such that it was internally consistent and capable of high inter-rater reliability. As soon as we started formal coding we saw immediately that real-world connections tended to occur in clusters, informing and reinforcing each other as students discussed or debated a particular point. The coding scheme underwent considerable revision, but reaching our goal of 80% inter-rater reliability proved to be difficult. Initially, we were evaluating   154 inter-rater reliability on a statement-by-statement basis. This led to the difficult and frustrating results where the two researchers would both code a cluster of real-world connections, but would code the individual statements within that cluster differently. At that time we were trying to only code “new” connections, and avoid coding anything that was a repeat of an earlier statement or idea. This meant that in circumstances where students built upon each others’ ideas, a difference in our judgment regarding which idea was the first would yield no agreement in our codes, even though both researchers had clearly identified that there were real-world connections emerging in the students’ interactions. The above problems were relieved when we shifted our perspective on what constituted a meaningful real-world connection. We realized that the reason that real-world connections tended to occur in clusters was that students engaged in collaborative group problem-solving were co-creating these connections and while an individual statement by an individual student might not encapsulate an entire idea, it could contribute to the ongoing evolving idea that was being developed by the group. With this new perspective of recognizing the importance of the students’ interaction with each other and the distributed nature of the real-world connections, we also started to see the importance of repetitions. When a student repeated a real-world connection that had been made by one of their peers they were often questioning it or reinforcing it, actions that had important meaning for the ongoing negotiation of that idea. Similarly, when a student repeated a real-world connection that they themselves had recently made, it often functioned as an emphasis to support their argument for that idea. Therefore, we agreed to change the scheme in 2 ways: 1. In recognizing the co-created and distributed nature of real-world connections, we would code with the individual statements as the unit of analysis but with a ± 6 second margin of error. This means that if one coder had coded a real-world connection, that code would be considered “good” if the   155 other coder had similarly coded a real-world connection within 6 seconds. Because the average length of a statement in the data set is approximately 3 seconds, this corresponds to a ±2 statement margin of error. 2. Recognizing the important social and argumentative value of repetitions in the context of an evolving group discussion, we code students’ repetitions of their own ideas or of others’ ideas when those repetitions were made in service of developing an ongoing argument or idea. Repetitions of earlier real-world connections that were merely reciting what had been decided earlier without challenging or adding anything to it would not be coded. 6.3.5.2 RWC Inter-Rater Reliability Testing The following table shows the Inter-Rater Reliability of all of the episodes coded during the coding scheme development.  Midway through the scheme development, it was necessary to switch to a different coding partner due to the beginning of a new term and Coder #1’s concomitant teaching responsibilities. Work with the next coder shows a steady increase in IRR until we reach a consistent performance of better than 80% IRR without significant changes to the coding scheme.  As with the development of the epistemological framing coding scheme, I judged this consistent performance of better than 80% IRR as evidence of sufficient reliability.   156 Episode  IRR CODER #1 1 78% 2 73% 3 89% 4 87% CODER #2 1 59% 2 71% 3 55% 4 81% 5 85% 6 87% Table 27:  Inter-rater reliability testing for Real-World Connections coding scheme.  The IRR was calculated as 1 - (number of RWC coded more than 6 seconds from the other coders’ RWC) / (total number of RWC coded). 6.3.5.3 Summary of RWC Coding Scheme As described above, the following categories of real-world connection emerged from our desire to find an internally consistent coding scheme that identified instances where students were synthesizing physics knowledge and real-world knowledge, or were making use of real-world knowledge in a meaningful way in their problem-solving. Although we came to recognize the interactional and distributed nature of these real-world connections, for the purposes of coding the unit of analysis was taken as a single statement. Firstly, because we are implicitly treating the real-world connections as positive uses of everyday resources in a physics context, any comments that are explicitly negative about the realism of the recitation problem were considered to be Negative RWC.  For example, when a student says “If we'd do this in real life, I think we'd just use common sense [instead of physics],” they are implying that physics is not useful in real life.  While the students are making a   157 connection between physics and the real world, this connection seems unlikely to contribute to a positive opinion of the relevance of physics to the real world and is therefore considered separately  (see section 6.4.5 below). Note that categorization as negative RWC was reserved for comments that were explicitly negative.  Many comments could have been interpreted as revealing students’ negative judgment of the realism of the tutorial, but this analysis would require in-depth analysis of the function of those comments within the broader conversational context which is a level of analysis to which this coding does not extend.  Rather, in order to maintain a fairly simple coding scheme that enabled high IRR and in order to assess a larger corpus of data, I chose to make this judgment of Negative RWC only on those statements where the students made their judgments explicitly. After identifying the explicitly negative RWC, five major categories of real- world connection were identified, listed in Table 28 below. Each category is listed with an Archetype statement, which is an idealized example of the kind of statement that would be given that particular code.  These categories are described in more detail in the following sections, and the full coding rubric is available in Appendix E.   158 RWC Category Definition Archetype Assumption Use of RW resources to define mathematical relationships “Because of X in the real world, we should use value Y in our calculation” Interpretation Use of RW resources to interpret meaning of abstract forms “These abstract or mathematical forms must mean X”  (because of my knowledge of similar situations) Evaluation Use of RW resources to evaluate plausibility of results “Our answer can’t be right because that would have unreasonable consequences” Personalization Use of RW resources to illustrate situation described in tutorial “If I were in this situation, this is what I would do.” Meta-statement Use of RW resources to comment on realism of recitation problem itself “I doubt that anyone would ever do this.” Table 28:  Types of real-world connection identified during collaborative group problem- solving in introductory physics. Early in the development of the coding scheme, the two researchers would code each instance of real-world connection by what type it was in the taxonomy. We discovered that it was very common for us to disagree on the type of real-world connection, even when we had very high agreement on which statements were real-world connections.  This is a clear indication that the following taxonomy does not represent non-overlapping categories. However, this list is still valuable in that two researchers working from this taxonomy can achieve high inter-rater reliability in identification of statements that meet one or more of these categories. ASSUMPTION The first category of real-world connections is Assumptions. This refers to instances where students use their knowledge of everyday relationships and quantities in order to define quantities or mathematical relationships within their model. The context-rich problems used in Physics 100 are designed to require students to make some assumptions in order to solve them. These assumptions can be anything from students assuming the speed of a car is 50   159 km/h to assuming that the airbags don't deploy during a crash, which would affect the force exerted on a passenger by their seatbelt. Because not all assumptions that students make are rooted in their knowledge of the real world, assumptions which are not explicitly connected to a real world situation and are mathematically convenient are not coded.  An example of this type of bald assumption