Science, Faculty of
Earth, Ocean and Atmospheric Sciences, Department of
DSpace
Clark, Miranda
Szilagy, Jordyn
Fitzpatrick, Zara
Despite the increasing prevalence of misinformation in many documentary films today, film festivals have not developed a systematic way to verify their accuracy and credibility. This gap in the film selection process can compromise the integrity of science communication and reduce public trust. To address this issue, we (three UBC Environmental Science students) partnered with the British Columbia Environmental Film Festival Society (BCEFF) to develop an advanced film selection framework that ensures selected films are scientifically sound and present minimal bias. We drew from existing fact-checking processes and consulted with experts in the film industry, such as filmmakers, producers, directors, and festival organizers. These consultations ensured the framework would be practical, feasible, and aligned with the industry’s needs. The core components of this framework are a filmmaker questionnaire, a scoring system, and detailed handbooks to guide the process. These tools were trialled on past films submitted to BCEFF with internal mock evaluations, peer evaluations with students from the University of British Columbia, and evaluations from film industry experts. The results demonstrated a high consistency across different reviewer groups, indicating that the framework is effective in assessing the film’s accuracy. However, the trials also revealed a need for clearer definitions and refinements within the tools to further enhance consistency. The final robust film selection framework provides an effective and simple way to evaluate and screen films for misinformation. It serves as a foundational tool that other science film festivals can adapt and build upon to enhance the accuracy and credibility of their program. The advanced film selection framework will be piloted in BCEFF’s 2025 film festival to determine the efficacy and efficiency of our framework in a live festival scenario. This framework has the potential to create broader systemic change within the film industry, encouraging filmmakers to adopt higher standards of scientific accuracy and fostering a culture of responsible storytelling.
DEVELOPING AN ADVANCED SELECTION FRAMEWORK FOR FILM FESTIVALS By: Miranda Clark, Jordyn Szilagy, and Zara Fitzpatrick ENVR 400, Environmental Science Film Festival In Collaboration with the British Columbia Environmental Film Festival and the University of British Columbia (UBC) May 20, 2025 2 Abstract Despite the increasing prevalence of misinformation in many documentary films today, film festivals have not developed a systematic way to verify their accuracy and credibility. This gap in the film selection process can compromise the integrity of science communication and reduce public trust. To address this issue, we (three UBC Environmental Science students) partnered with the British Columbia Environmental Film Festival Society (BCEFF) to develop an advanced film selection framework that ensures selected films are scientifically sound and present minimal bias. We drew from existing fact-checking processes and consulted with experts in the film industry, such as filmmakers, producers, directors, and festival organizers. These consultations ensured the framework would be practical, feasible, and aligned with the industry’s needs. The core components of this framework are a filmmaker questionnaire, a scoring system, and detailed handbooks to guide the process. These tools were trialled on past films submitted to BCEFF with internal mock evaluations, peer evaluations with students from the University of British Columbia, and evaluations from film industry experts. The results demonstrated a high consistency across different reviewer groups, indicating that the framework is effective in assessing the film’s accuracy. However, the trials also revealed a need for clearer definitions and refinements within the tools to further enhance consistency. The final robust film selection framework provides an effective and simple way to evaluate and screen films for misinformation. It serves as a foundational tool that other science film festivals can adapt and build upon to enhance the accuracy and credibility of their program. The advanced film selection framework will be piloted in BCEFF’s 2025 film festival to determine the efficacy and efficiency of our framework in a live festival scenario. This framework has the potential to create broader systemic change within the film industry, 3 encouraging filmmakers to adopt higher standards of scientific accuracy and fostering a culture of responsible storytelling. 4 The Authors Figure 1. Photo of our team. Jordyn Szilagy on the left, Zara Fitzpatrick in the middle, and Miranda Clark on the right. Miranda Clark – Miranda is a 4th year Environmental Science student at the University of British Columbia (UBC) specializing in sustainability. Within her degree, she has worked for Environment and Climate Change Canada (ECCC) for two co-ops, employing her with relevant experience in expert consultations, framework development, and data analysis. Moreover, she is currently volunteering with the BCEFF as a film reviewer and understands the context of the films that will be fact-checked. Jordyn Szilagy – Jordyn is a 4th year Environmental Science student at UBC, concentrating in the sustainability field. Some of her skills that will help throughout this project include research, writing, teamwork, and communication skills. These skills have been developed through completing foundational environmental courses at UBC, retail experience, and a previous internship in Greece. 5 Zara Fitzpatrick – Zara is a 4th year student studying Environmental Science with a concentration in Sustainability. During her time at UBC, she has developed critical thinking and data analysis skills from core environmental courses ENVR 200 and ENVR 300. Moreover, she has previously engaged in initiatives involving environmental sustainability through work experience with Earthwise Society, a sustainable agriculture non-profit in which she also gained system management skills. 6 Table of Contents Abstract ....................................................................................................................... 2 The Authors ................................................................................................................ 4 1.0 Background Information ........................................................................................ 8 1.1 Misrepresentation and Bias .......................................................................................... 8 1.2 The Problem ................................................................................................................. 9 1.3 The Current Film Selection Process ............................................................................ 10 1.4 Fact-checking Tools ................................................................................................... 11 1.5 Our Solution ............................................................................................................... 12 2.0 Methods .............................................................................................................. 12 2.1 Pre-Interviews ............................................................................................................ 12 2.2 Rubric ........................................................................................................................ 13 2.2.1 Rubric Overview .......................................................................................................................... 13 2.2.2 Rubric Format .............................................................................................................................. 14 2.2.3 Internal Mock Rubric Evaluation .................................................................................................. 16 2.2.4 Peer Mock Rubric Evaluation ....................................................................................................... 17 2.2.5 Internal Expert Driven Rubric Evaluation ...................................................................................... 18 2.3 Questionnaire ............................................................................................................ 18 2.3.1 Questionnaire Overview .............................................................................................................. 18 2.3.2 Internal Mock Questionnaire Completion .................................................................................... 19 2.3.3 Expert Mock Questionnaire Completion ...................................................................................... 19 2.4 Handbook .................................................................................................................. 20 2.4.1 Handbook Overview .................................................................................................................... 20 2.4.2 Checkable Facts .......................................................................................................................... 20 2.4.3 Key Facts ..................................................................................................................................... 21 2.4.4 Reference Sources ...................................................................................................................... 23 3.0 Results ................................................................................................................ 24 3.1 Filmmaker Questionnaire and Handbook .................................................................... 25 3.2 Film Reviewer Scoring Rubric and Handbook .............................................................. 27 4.0 Discussion .......................................................................................................... 28 4.1 Our Tools .................................................................................................................... 28 4.2 Implications ............................................................................................................... 29 4.2.1 Competitive Edge ........................................................................................................................ 29 4.2.2 Incentivizing Truth........................................................................................................................ 30 4.3 Challenges ................................................................................................................. 30 5.0 Next Steps ........................................................................................................... 31 5.1 Training BCEFF with New Framework .......................................................................... 31 5.2 Moving Forward - Pilot Project ..................................................................................... 31 7 6.0 Conclusion .......................................................................................................... 32 Acknowledgements .................................................................................................. 32 References ............................................................................................................... 33 Appendix A: Form Questions ..................................................................................... 38 Appendix B: Film Experts Contributions .................................................................... 39 Appendix C: Adaption of our Tools from Feedback .................................................... 39 Appendix D: Final Handbooks ................................................................................... 41 Appendix E: Step By Step Guide to BCEFF Filmmaker’s Questionnaire ...................... 70 Appendix F: Simplified Process of using Fact-Checking and Bias BCEFF Reviewing Tool .......................................................................................................................... 71 8 1.0 Background Information 1.1 Misrepresentation and Bias In recent years, some documentary filmmakers have focused their films on appealing to consumers’ interests and desires to create emotional investment from the viewer instead of staying to the true story (Eitzen, 2018). These filmmakers capitalize on the shock value of misinformation and strong narratives that can better views (Eitzen, 2018). This has led to a greater spread of misinformation and the sharing of, intentional or unintentional, inherited biases through films. One example of the dangers of misrepresentation and bias is the Netflix documentary “Seaspiracy” where the film misrepresented information involving fisheries, leading to wrongful activism and harm to fishing industries and communities (Harris, 2022). In environmental films, the concept of the “funding effect” is also prominent as it impacts a film’s bias. This term was first used in the pharmaceutical industry, where study outcomes correlated with funding sources (Krimsky, 2012). Specifically in environmental films, an example of this effect would be if an oil company sponsored an environmental film, subtly enforcing the industry’s preferred narrative by downplaying the reality of climate change. Unfortunately, this can lead to film festivals screening and sharing films that may spread misrepresented information or biases. Mitigating these challenges can help ensure that film festivals remain a trusted platform for screening accurate films. Misinformation and bias may present significant challenges to public understanding and the actions that follow. Misinformation and bias within media, including documentaries, can influence an individual’s beliefs, decisions, and behaviours (Ecker et al., 2024). Filmmakers are often perceived by the public as trusted experts. Thus, their portrayals can negatively impact 9 societal views on species or the environments they showcase (Williamson et al., 2022). For example, one well-known group of organisms facing these challenges is sharks. In documentaries like Shark Week, sharks are frequently framed as violent predators through misleading narratives and selective editing. These portrayals not only distort public perception but can also lead to misinformed policy decisions with harmful consequences for shark conservation (Williamson et al., 2022). Film misinformation has broader implications, potentially shaping societal discourse and resulting policies. It is, therefore, essential to recognize and mitigate these challenges to support a more accurate and responsible documentary film industry. 1.2 The Problem The British Columbia Environmental Film Festival Society (BCEFF) was founded in 2021 by Jennifer Jackson, Pezhman Hadavi, and Maziyar Sharafabadi to bridge the gap between science and film. By providing a platform for environmental storytelling, BCEFF enables people without a background in science to be educated and informed on complex environmental issues. BCEFF has gained significant popularity since the festival's launch in 2021, where they receive over 200 films annually from diverse filmmakers. However, the increasing volume of submissions brings new challenges, particularly the growing risk of misrepresenting information and the incorrect portrayal of scientific topics. Due to this, BCEFF was seeking a more streamlined and scalable approach to evaluate both the validity of facts and bias in submitted films. 10 1.3 The Current Film Selection Process The current film selection process for BCEFF (Figure 2) lacks a robust methodology to check the validity of facts and identify biases in films. Films are submitted along with responses to a few preliminary questions; however, no questions are asked to target potential biases or factual content. Figure 2. BCEFF’s current film-reviewing process Volunteers then review each film for engagement and rate them out of 10 based on various categories, such as writing and cinematography (Figure 3). If the films are deemed engaging, scoring over 7, they are scientifically reviewed by Jennifer (BCEFF founder and scientist) to evaluate factuality. This process was time-consuming and inefficient, requiring the 11 film selection team to sift through numerous films and research scientific content for each one. Moreover, the process was not detailed enough to thoroughly examine the facts and biases. Figure 3. The current scoring sheet for engagement that is used by volunteers at BCEFF 1.4 Fact-checking Tools There is no available published research specifying the process of verifying the scientific accuracy of films or documentaries. However, a few film screening platforms have devised systems to minimize misinformation. CBC mandates filmmakers to submit an annotated script, citing facts with three peer-reviewed references (CBC, 2024). Similarly, National Geographic directs filmmakers to submit annotated scripts to be verified; while assessing any conflicts of interest apparent in funding sources (National Geographic, 2013). While these simple processes exist, they are not well utilized across the film industry. Conversely, in general media, there are several existing fact-checking frameworks, including software and artificial intelligence (AI) websites, human-powered organizations, and 12 art-applicable peer-reviewed processes (Esteban-Navarro et al., 2021; Glisovic et al., 2016; Graves., 2017; Hight., 2022; La Barbera et al., 2023). These sources have developed key criteria for fact-checking inaccurate media and art. However, there is ongoing debate on the best methodology, as there is a clear trade-off between the efficiency and validity of existing frameworks. 1.5 Our Solution Our objective was to develop an improved film selection framework to minimize bias and misinformation. Given BCEFF’s limited resources, we aimed to put more of the onus on filmmakers to demonstrate the quality of their films. Additionally, we wanted to establish a standardized approach to ensure that films are evaluated consistently, regardless of who is on the film selection team. Drawing from existing literature and discussions with film industry experts, we developed a questionnaire for filmmakers and a rubric that the BCEFF reviewing team will use to evaluate the film. Additionally, we prepared two handbooks to guide filmmakers and reviewers as they apply our framework. 2.0 Methods 2.1 Pre-Interviews Before developing our selection framework, we conversed with experts in the film industry, including Carolyn Whittaker, Mike Douglas, and Anny Tubbs (Appendix B). BCEFF connected us with these specialists to help us better understand the context and diverse perspectives surrounding this complex issue. We asked various questions to understand the expert’s firsthand 13 opinions and experience regarding factual content, biases, and funding sources. These discussions guided the preliminary development of our framework. The questions included: • Do you find it challenging to accurately depict the true story? • How much research do you do before versus during filmmaking? • Do you find a lot of misrepresentation/inaccuracy of information within the film industry? • How much do inherent biases affect the filming process? Do you recognize biases in yours or other films, or try to get the full picture? • Do you ever get help with fact-checking from people with more knowledge on a topic? • Do you experience struggles with sources of funding? • How transparent do you find films are in terms of funding sources? 2.2 Rubric 2.2.1 Rubric Overview To score films for misinformation and bias, we used a method designed to ensure consistent evaluations across a wide range of films and reviewers. To achieve this, we developed a rubric. Generally, rubrics outline key assessment criteria and define specific standards for how well each criterion should be met (Jonsson & Svingby, 2007). We used an “adopt and adapt” method to create our robust rubric (Crusan., 2010). This meant creating a draft rubric and then adapting it based on trial assessments and consultations with experts (Clauser, 2000). This is a common way to adapt rubrics, allowing experts to provide input to produce scoring guidelines in collaboration with the rubric developers (Bejar. 1991). 14 2.2.2 Rubric Format We conducted a literature review and communicated with key experts (Appendix B) to distinguish important categories and content areas of the rubric. We implemented seven categories for key components that filmmakers would be scored on (Table 1). Table 1. Key components to evaluate films for misinformation and mitigate bias Key Components Description Importance Provided Facts This component measures if the filmmaker has provided all their key facts from the film, highlighted in an annotated script or extracted into their questionnaire response. Being transparent about what is a fact prevents misleading storytelling (Lewandowsky et al., 2012). Facts Cited This component measures if the filmmaker has cited all the key facts that they provided. Filmmakers providing a fact from their film should have a reference to support the claim. Referencing the fact shows that it comes from a verifiable source. Validity of References This component measures the creditability and reliability of the source cited. A reference should be from a reliable and credible source. 15 Conflict of Interest This component measures if the films content has any personal or professional stake in the matter. Identifying conflicts of interest ensures films that are screened aren’t biased or agenda driven. Funding Sources This component measures where the money comes from to finance a film. This can include corporate sponsorships, private companies, industry stakeholders, grants, crowdfunding, or philanthropic foundations. Funding from certain organizations can create bias especially when communicating science information (Drucker et al., 2016). Transparency of the funding source allows reviewers to assess whether it may have shaped the main message of the film. Intellectual Property This component measures who owns the intellectual property rights of the film. Important to verify ownership and who controls how film is used. Balanced Perspective This component measures how well the film covers different viewpoints or different sides of the story. Presenting only one side of an argument can mislead audiences. This category reminds filmmakers the importance of acknowledging the other side to the story. 16 After trialing different options, we chose a 5-point scoring system for the rubric. Fewer points didn’t allow for enough differentiation, leading to clustered scores. On the other hand, too many points made it difficult to distinguish between levels. We also developed a failure threshold for each category such that if a film scored below the threshold in any single category, it would automatically fail and not proceed to the next stage of film selection. Some experts we consulted suggested setting a very high threshold; however, we do not want this tool to discourage filmmakers from submitting their films. In Figure 5, the final failure threshold is indicated by a red line. Any film that scores a one and below will fail except for the category's validity of references and balanced perspective (Table 1). For these two categories, films would only fail if they scored a zero to create more leniency in these two challenging components. For the category validity of references, we anticipate that not all filmmakers would have science backgrounds and strong skills in sourcing non-biased references. Moreover, for the balanced perspective category, we found that the nature of telling a story often involves portraying a one-sided perspective. While we acknowledged the importance of recognizing the whole story, we also wanted to maintain the film’s engagement, thus lowering the failure threshold. For the first year of adaptation to BCEFF, we feel this approach would be most effective and will be adapted in the future. 2.2.3 Internal Mock Rubric Evaluation To evaluate the rubric, each member of our team of three created a mock questionnaire for a film submitted to the 2024 BCEFF. We swapped completed questionnaires with each other and evaluated the film with the rubric. We each scored the film on the rubric based on our interpretation of the film alongside the completed questionnaire, using the film reviewer handbook as a guide. This process was done twice, with each student ‘reviewing’ two films. 17 In this comparative analysis, all three films were scored identically by all three of us in the scoring rubric. This signified a good start in terms of the consistency and effectiveness of our tools, although we anticipated that our understanding would differ from general reviewers and filmmakers without our expertise. 2.2.4 Peer Mock Rubric Evaluation To gain independent feedback on the scoring rubric, we invited other UBC students to evaluate the usability, efficacy, and general comprehensibility of the rubric and handbook. One of the three films and associated mock questionnaire was sent to nine students (primarily peers from ENVR 400). Each student was instructed to watch the film and use the handbook to score the film on the rubric. They were also instructed to submit feedback via a feedback form outlining the key successes and challenges of the tool (questions included in Appendix A). The peer trial determined that our rubric and handbook were effective in guiding reviewers to fill out the rubric correctly. Most peers’ rubrics scores were almost identical to our anticipated results, with only one or two minor discrepancies in points. These discrepancies could be explained by a lack of detailed definitions and not enough examples in the film reviewer handbook (feedback and applied changes found in Appendix C). For short films (less than 10 minutes), students stated that the entire scoring process took 15-30 minutes. This indicated that it was not an overly onerous process and could be completed in a timely manner with no previous experience. 18 2.2.5 Internal Expert Driven Rubric Evaluation We used the film industry expert’s completed questionnaire (as discussed in section 2.3 below) to score their films using the handbook and rubric. We scored three films: Animal Pride (2024) by Carolyn Whittaker, Sam and Me (2021) by Mike Douglas and Trapped in Plastic (2023) by Anny Tubbs. We individually evaluated the films on the scoring rubric, noting any discrepancies we found while following the evaluation process. All three of us scored the films consistently across all rubric categories except for the balanced perspective category. We realized that our explanation of this category was prone to misinterpretation and would need to be drastically revised. The trial also highlighted the need to clarify terms in our handbook, such as ‘provided fact,’ ‘cited facts,’ and ‘conflict of interest.’ Additionally, the trial revealed the unanimous step-by-step processes we all followed when reviewing and evaluating the films, reminding us to include a detailed procedural guide for film reviewers in our handbook. 2.3 Questionnaire 2.3.1 Questionnaire Overview In addition to the rubric, we created a questionnaire to allow filmmakers to share more detailed insights about their submissions beyond what could be understood from viewing the documentary alone. The questionnaire would be used to collect standardized information through a structured set of questions (Malhotra, 2006). The questionnaire we created helped analyze the factual integrity and biases within documentaries. The prompts in the questionnaire were created from discussions with experts in the field (Appendix B) and our literary research. It focuses on 19 the seven key categories outlined in Section 2.2.2 and is designed to gather deeper insights into these areas. 2.3.2 Internal Mock Questionnaire Completion Each member of our three-person team viewed one short (less than 20 minutes) film submitted to BCEFF in the 2023-2024 film festival. After watching the film, we each filled out a mock questionnaire based on the facts presented in the film, cross-referencing available sources to verify the facts and guessing funding sources and potential biases. We applied varying levels of rigor while completing the questionnaire (i.e., one reflected a well-completed questionnaire, while another reflected a poor/incomplete questionnaire response). This step allowed our team to view the questionnaire and associated filmmaker handbook from a filmmaker perspective, as well as provide the primary resources needed for the following trials. From these trails, we found that we needed to add more detailed definitions and examples of checkable and key facts. Specifically in targeting controversial statements that may seem anecdotal or spoken within the fictional plot of a film. These examples were added to the updated user manual and questionnaire. 2.3.3 Expert Mock Questionnaire Completion Blank questionnaires were sent via SurveyMars to three filmmakers/industry experts with the corresponding filmmaker handbook. They were asked to complete the questionnaire using one of their films and submit their feedback via a feedback form (questions included in Appendix A). Following these trials, we met with each expert to better understand their insights and suggestions for the draft tools. 20 The filmmakers provided valuable insights during the discussions that we incorporated into our framework (feedback found in Appendix C). They highlighted the importance of clearly conveying the intent behind our advanced film selection process to inspire and guide filmmakers. Additionally, they recommended providing context on how questionnaire results will be evaluated, such as including the scoring rubric and explaining our failure threshold. With these updates, our goal is to ensure our questionnaire will inspire and not discourage filmmakers. 2.4 Handbook 2.4.1 Handbook Overview We developed two handbooks to assist filmmakers and film reviewers in using and evaluating our rubric and questionnaire, tailored to the respective audience (filmmaker vs film reviewer). The filmmaker handbook provides detailed guidance on completing the questionnaire including additional prompting questions and clarifications of key terms. The film reviewer handbook offers step-by-step instructions for analyzing the film and assigning points to the different components of the rubric. 2.4.2 Checkable Facts To effectively verify facts presented in films, it was also essential to determine which statements can be fact-checked (Merpert et al., 2017). According to Chequeado (2015), checkable facts can be classified into four categories: historical data, comparisons, legality or statistical statements, to help identify verifiable claims. In contract, research has found that opinions, anecdotal statements, generalized value statements, and some forms of future projections cannot be verified and, therefore, do not qualify as ‘checkable facts’ (Barnwal et al., 2019; Chequeado., 2015; Graves., 2017; Merpert et al., 2017). It is also important to 21 acknowledge that Indigenous ways of knowing cannot be considered ‘checkable’ using this Westernized approach. With that in mind, Indigenous knowledge will not need to be verified. Evidently, defining checkable facts is not a simple process; however, the criteria outlined above can help guide the process (Merpert et al., 2017). Culminating the research, various classification systems, and expert’s input, we developed the following criteria to identify checkable facts in films. Checkable facts would often include the following qualities: • Quantitative data - measurable information (including statistics or other numerical information) • Objective statements - based on facts (not opinions or anecdotal evidence) • Testable claims - could be verified using data, records, or observations Non-checkable facts often include the following qualities: • Anecdotal or Indigenous evidence • Clear opinions or guesses - statements with the words “think”, “likely”, “probably”, “in my opinion” • Facts stated within a clear fictional ‘story’ of a film 2.4.3 Key Facts While rigorous fact checking is important, film selection teams and producers have limited time to cite or verify every checkable statement. The International Fact Checking Network (n.d) instructs certified fact checking platforms to follow a two-step processing when choosing what statements to verify: step 1) can the statement be fact checked and step 2) does the statement need to be fact checked. For example, PolitiFact, an independent media fact checking platform in 22 the United States filters for crucial statements by considering the following questions (Holan., 2018): • Is the statement rooted in a fact that is verifiable? • Does the statement seem misleading or sound wrong? • Is the statement significant? • Is the statement likely to be passed on and repeated by others? • Would a typical person hear or read the statement and wonder: Is that true? Moreover, Boom, an India and South Asian fact checking platform, selects based on similar criteria (Boom., n.d): • Is the statement newsworthy? • Does it impact many people? • Is the content provocative or sensitive? • Could it cause harm if left unchecked? Australia’s RMIT University and ABC media’s combined fact checking system follows a similar regime, verifying statements that would make one think “can that be right?” (RMIT University & ABC., n.d). While this selective framework enhances efficiency and allows large media fact-checking platforms to optimize their resources, it lacked the rigor that film festivals strive for. To balance stringency with a more time-efficient approach, we have advanced PolitiFact’s methodology to identify and prioritize “key facts” while introducing a lower threshold to determine what are not key facts (Graves., 2017; Holan., 2018; International Fact-Checking Network., n.d.; Uscinski & Butler, 2013). In our framework we have identified key facts based on the following conditions: 23 Key facts as likely to raise some of the following questions: • Does the fact sound wrong or seem misleading? • Is the fact significant? • Is the fact likely to be passed on and repeated by others? • Would a typical person hear or read the statement and wonder: Is that true? Key facts are not: • Well-known accepted truths or general knowledge (i.e “turtles lay eggs”) • Trivial or unrelated details to the main message of the story (i.e “there are four people in the birdwatching team”) • Excessive technical information without relevance • Redundant facts previously stated in a different way and with no new insights 2.4.4 Reference Sources Filmmakers often rely on numerous resources to gather information for films, especially since not all of them are scientists. High-quality references are important when trying to mitigate misinformation and biases. In the case of science films, we found that high quality sources can come in many different forms. Some include academic publications, reputable news organizations, experts within a profession and government websites (CBC, 2024; National Geographic, 2013). Academic papers are often viewed as highly credible as they are peer-reviewed and generally have authors with subject matter expertise. However, filmmakers may not always have access to peer-reviewed literature or comprehensive information on every fact requiring a citation. To address this, we identified reputable organizations using the Ad Fontes Media Bias Chart (Ad Fontes Media, 2025). Additionally, we have included in two handbooks 24 multiple other sites including government sites to illustrate potential good references in addition to peer reviewed literature. Assessing whether certain peer-reviewed journals are reliable can be difficult for even the most proficient scientists. One tool that we found to be beneficial is the impact factor of a journal. These scores are available via Journal Citation Reports (JCR) on Web of Science and Cite Score on Scopus. This impact factor can be used to indicate the status of journals through the average number of citations its articles receive (Sharma et al., 2014). There is a level of caution to take with applying impact factors as they only rely on the number of citations, however, it does provide a reference point for journal credibility. We have included the impact factor resource in both of our handbooks to be referred to by both filmmakers and reviewers to provide further clarity how this resource can be useful in determining the validity of peer-reviewed journals. Many filmmakers acquire knowledge from news outlets; thus we outlined the biases that could come from each outlet. Biased news outlets can shape readers’ perspectives by selectively presenting information and framing narratives, often through selection bias and narrative bias (Galeazzi et al., 2024). This emphasized the importance of critically evaluating potential biases in news articles in our framework. Understanding the biases that come with specific news outlets and using resources such as https://ground.news, can help ensure that the information being delivered is credible, reliable, and impartial. 3.0 Results With the valuable insights from our mock trials, and discussions with film industry experts, peers, and the BCEFF team, we developed our final advanced film selection framework. 25 Our framework incorporated two critical tools, the filmmaker questionnaire, and scoring rubric, as well as the corresponding specialized handbooks to guide both filmmakers and reviewers throughout each process. The framework expands on BCEFF’s original selection process (Figure 2), incorporating our key enhancements (Figure 4). Figure 4. The Advanced Film Selection Framework 3.1 Filmmaker Questionnaire and Handbook The filmmaker questionnaire is available on SurveyMars to ensure straightforward completion and results from filmmakers (Filmmaker Questionnaire). In the questionnaire, there are two components for filmmakers to complete. First, there is a set of questions for filmmakers to answer. Second, filmmakers are asked to provide a list of key facts presented in their film. To do so, filmmakers can either upload an annotated script with the sources of key facts within the film or input facts into a fillable table provided including the timestamp and at least one reference source. 26 The finalized questionnaire asks the following questions: • What is the title of your film submitted to BCEFF? • What is your name and role in producing your film? • Who owns the rights and intellectual properties of this film? • What funders were included in financing your film production? Include any relevant details. • Did any of the funders listed in the question above have any editorial involvement (i.e. did they provide notes on any of the film content that adjusted the film, co-producers or executives came from commercial companies)? • What inspired you to make this film, and what message do you hope to convey? • Do you have any personal or professional relationships with companies or organizations presented in the film? • Are there any potential biases in the presentation of your material you’d like to disclose that were not covered above? • Is there anything you’ve done to mitigate biases in your film? • Do you feel you film portrays a balanced perspective that includes important content? The designated table available for filmmakers to provide key facts appears as the following: Table 2. Key facts format Fact Cite Source Verifying Fact Time Stamp (min:sec) Fact 1 Fact 2 27 Fact 3 The filmmaker questionnaire is designed to be used in conjunction with our filmmaker handbook. The handbook guides the filmmaker through the process of completing the questionnaire, understanding which facts to cite, how to properly reference statements, and answer the remaining questionnaires with ease and clarity. The handbook is available as a link in the survey to ensure easy access to this critical resource and linked here: Filmmaker Handbook . The filmmaker handbook is available in full in Appendix D. To provide clarity and an understanding of the process of the questionnaire, we created an easy-to-read step-by-step guide for filmmakers to review before answering the set of questions, which can be found in Appendix E. 3.2 Film Reviewer Scoring Rubric and Handbook The finalized rubric displayed below (Figure 5) includes the seven critical factors and levels used to assess the factual accuracy and bias components of each film. 28 Figure 5. The rubric used to score films based on factual accuracy and biases. The failure threshold is represented by the red line, whereby if in any category a film is scored below it will automatically be eliminated from the selection process. Alongside the scoring rubric, film reviewers also have access to a reviewer handbook. This handbook serves as a critical guide to ensure all reviewers evaluate films with the same level of scrutiny, detail, and understanding. The handbook includes our step-by-step process of evaluating films in addition to clarifying the nuances of each scoring level. The film reviewer handbook is included in Appendix E and via this link: Film Reviewer Handbook. We also created a simplified overview of how to use our tools, allowing BCEFF reviewers to quickly understand the process before applying the rubric. This can be found in Appendix F. 4.0 Discussion 4.1 Our Tools Our team has developed an advanced film selection framework using a filmmaker questionnaire, a scoring rubric, and corresponding handbooks for both the questionnaire and rubric to streamline the BCEFF documentary submission process (Figure 4). Our framework strives to ensure that selected films are factually accurate with minimized bias. The tools have been honed based on peer-reviewed literature, discussions with experts, and feedback from both film industry experts and UBC student peers. The framework will help BCEFF ensure that all films selected and screened will present accurate factual content, have a high degree of objectivity, and minimal bias. This will increase the transparency and trustworthiness of environmental storytelling and foster greater public understanding and knowledge of environmental issues through BCEFF. 29 4.2 Implications Through the creation of our framework, we aim to set a new standard that film festivals can adopt to improve credibility. The framework will help ensure that films viewed by the audience are both compelling and credible, fostering a more informed audience. In turn, this could lead to an increase in public engagement, better decision-making skills for viewers, and stronger support for initiatives, such as policy changes. 4.2.1 Competitive Edge By creating a successful framework to evaluate films for scientific accuracy and biases, we hope to also give BCEFF a competitive edge. This may increase their performance as a film festival while pressuring other film festivals to improve screening methods to ‘keep up’ with competition in the industry. Studies have found that corporate social responsibility, corporate reputation, and innovation play a large role in the success of various organizations (Le., 2023; Ruiz-Ortega et al., 2016). Advertising BCEFF as the first ‘misrepresentation-free’ film festival will likely improve the corporate and innovative reputation of the festival leading to greater success. Moreover, studies have highlighted the prevalence of innovation mirroring and other forms of innovation uptake in the industry (Høegh-Guldberg, 2018). By developing our innovative system, we expect other film festivals will mirror and adapt the framework. We hope this will increase the factual accuracy in many film festivals thus incentivizing the film industry to reduce misrepresentation. 30 4.2.2 Incentivizing Truth Developing a rigorous, inclusive screening process for BCEFF may incentivize filmmakers to reduce biases in their films. Gentzkow and Shapiro (2006) found that when ‘companies’ are evaluated for misinformation (with the results available to the public), they are disincentivized to share misinformation. While BCEFF does not plan to publish film’s inaccuracies, they will select films with high factual validity and minimal biases, potentially with a verification stamp. Connecting to Gentzkow and Shapiro’s research (2006), we anticipate that our advanced film selection process may incentive filmmakers to choose non-biased funding sources or narratives, when possible, maintain editorial power in their film production, and consider the role of biases in their film’s narrative to maintain their competitive edge. 4.3 Challenges With our advanced framework, we have acknowledged and considered potential challenges for BCEFF. Film industry experts have emphasized the potential loss of film submissions to the BCEFF. With our rigorous evaluation system asking for funding sources, factual sources, and biases, BCEFF might start to build a reputation within the filmmaker and film festival community as overly exclusive. This reputation could unintentionally alienate independent filmmakers and audiences if the review process is perceived as excessively rigorous. As a result, filmmakers may be drawn to other festivals that seem more accessible and inclusive. It is critical to balance factuality, biases, and inclusivity with all future film selection processes to avoid backlash from the film community. 31 5.0 Next Steps 5.1 Training BCEFF with New Framework As with any new tool or system, proper training is essential to ensure it is used effectively and efficiently. Our framework will serve as a new evaluation mechanism for BCEFF when reviewing film submissions. To ensure that all reviewers understand our new process, our team plans to host a workshop to explain all aspects of the framework. This includes the filmmaker’s questionnaire, reviewer rubric, and the corresponding handbooks, all designed to work together to promote consistent evaluation, clear communication, and fair assessments. We will highlight the seven key elements our tools assess, emphasizing their importance for the festival and the broader community. While our trial runs have found the process to be intuitive when used alongside the handbooks, the workshop will provide an opportunity for reviewers to ask questions, offer feedback, and suggest potential improvements. 5.2 Moving Forward - Pilot Project Our advanced film selection framework will be piloted in BCEFF’s 2025 film festival to determine the efficacy and efficiency of the process in a live festival scenario. Filmmakers will be presented with the opportunity to complete the questionnaire when they are selected as the top 40 films; however, it will not be mandatory. The film reviewer team will then score the films using the handbook guide to do so. A survey will be sent to filmmakers and film reviewers involved in this pilot project to evaluate the time taken to complete the task, any challenges and uncertainties experienced, suggestions, and other relevant comments in applying our framework. We will also analyze factors such as how many filmmakers completed the questionnaire, the passing rate of films, and discrepancies in scoring patterns between reviewers. 32 Following the pilot project, we will continue to work with BCEFF to publish a paper outlining the effectiveness and simplicity of our tools using quantitative data analysis to support our findings. We hope this paper can be shared, applied, and adapted by other film festivals to combat the challenge of misinformation on a large scale. 6.0 Conclusion The tools we developed to enhance the film selection framework for the British Columbia Environmental Film Festival strive to effectively and efficiently minimize biases and misinformation that may be present in submitted films. Drawing on peer-reviewed research, expert consultations, and feedback from both experts and peers, we developed a questionnaire, rubric, and two corresponding handbooks. These new resources aim to standardize the evaluation process, ensure transparency, and increase the credibility of BCEFF. As we implement and test our tools in the 2025 festival, we aim to gather insight and feedback from filmmakers and reviewers to refine our tools and improve the framework. Through the creation and revision of our advanced film selection framework, we seek to set a new standard that film festivals can adopt to improve their credibility. The advanced framework will enhance fact-checking and bias processes, ensuring that film festivals uphold higher standards of accuracy and transparency. Long-term, these efforts will ensure that science-focused films are not only compelling but also credible, fostering a more informed population. Acknowledgements We would like to thank Jennifer Jackson, Tara Ivanochko, Pezhman Hadavi, Carolyn Whittaker, Anny Tubbs and Mike Douglas for supporting us throughout this project. 33 References Ad Fontes Media. (2025, March 26). Home | Ad fontes media. https://adfontesmedia.com/ Barnwal, D., Ghelani, S., Krishna, R., Basu, M., & Ghosh, S. (2019). Identifying fact-checkable microblogs during disasters: A classification-ranking approach. Proceedings of the 20th International Conference on Distributed Computing and Networking, 389–392. https://doi.org/10.1145/3288599.3295587 Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76(4), 522–532. Boom. (n.d.). Boomlive Methodology: Fact-Checking and News Verification. BOOM. https://www.boomlive.in/methodology CBC. (2024). Producers Handbook and format guide. CBC Docs. https://www.cbc.ca/independentproducers/content/pdf/PassionateEye-Format-Guide.pdf Chequeado. (2015). Método [Method]. Retrieved from http://chequeado.com/metodo/ Clauser, B. E. (2000). Recurrent issues and recent advances in scoring performance assessments. Applied Psychological Measurement, 24(4), 310-324. https://doi.org/10.1177/01466210022031778 Crusan, D. (2010). Assessment in the Second Language Writing Classroom. University of Michigan Press/ELT. https://doi.org/10.3998/mpub.770334 Dierickx, L., Lindén, C. G., & Opdahl, A. L. (2023). Automated fact-checking to support professional practices: systematic literature review and meta-analysis. International Journal of Communication, 17, 21-21. https://ijoc.org/index.php/ijoc/article/view/21071 34 Drucker, A. M., Fleming, P., & Chan, A.-W. (2016). Research Techniques Made Simple: Assessing Risk of Bias in Systematic Reviews. Journal of Investigative Dermatology, 136(11), e109–e114. https://doi.org/10.1016/j.jid.2016.08.021 Ecker, U. K. H., Tay, L. Q., Roozenbeek, J., Van Der Linden, S., Cook, J., Oreskes, N., & Lewandowsky, S. (2024). Why misinformation must not be ignored. American Psychologist. https://doi.org/10.1037/amp0001448 Eitzen, D. (2018). The duties of documentary in a Post-Truth society. In Springer eBooks (pp. 93–111). https://doi.org/10.1007/978-3-319-90332-3_6 Esteban-Navarro, M., Nogales-Bocio, A., García-Madurga, M., & Morte-Nadal, T. (2021). Spanish Fact-Checking Services: an approach to their business models. Publications, 9(3), 38. https://doi.org/10.3390/publications9030038 Galeazzi, A., Peruzzi, A., Brugnoli, E., Delmastro, M., & Zollo, F. (2024). Unveiling the hidden agenda: Biases in news reporting and consumption. PNAS Nexus, 3(11). https://doi.org/10.1093/pnasnexus/pgae474 Gentzkow, M., & Shapiro, J. M. (2006). Media Bias and Reputation. Journal of Political Economy, 114(2), 280–316. https://doi.org/10.1086/499414 Glisovic, S., Berkeley, L., & Batty, C. (2016). The problem of peer review in screen production: Exploring issues and proposing solutions. Studies in Australasian Cinema, 10, 1–15. https://doi.org/10.1080/17503175.2015.1133262 Grammarly. (2024). Grammarly. https://www.grammarly.com Graves, L. (2017). Anatomy of a Fact Check: Objective Practice and the Contested Epistemology of Fact Checking. Communication, Culture and Critique, 10(3), 518–537. https://doi.org/10.1111/cccr.12163 35 Harris, H. E. (2022). Science vs. Sensationalism: Lessons for Science Communication in Fisheries from Netflix’s Seaspiracy. Fisheries, 47(4), 154–156. https://doi.org/10.1002/fsh.10727 Hight, C. (2022). Deepfakes and documentary practice in an age of misinformation. Continuum, 36(3), 393-410. https://doi.org/10.1080/10304312.2021.2003756 Høegh-Guldberg, O. (2018). Between company and network practices: Mirroring innovative ideas. Scandinavian Journal of Hospitality and Tourism, 18(3), 278–302. https://doi.org/10.1080/15022250.2018.1497305 Holan, A. D. (2018). PolitiFact | The Principles of the Truth-O-Meter: PolitiFact’s methodology for independent fact-checking. https://www.politifact.com/article/2018/feb/12/principles-truth-o-meter-politifacts-methodology-i/ International Fact-Checking Network. (n.d.). Methodology for political fact-checking. Poynter Institute. Retrieved March 2, 2025, from https://www.poynter.org/courses/fact-checking-fundamentals-with-ifcn/lessons/what-is-fact-checking-2/topic/methodology-for-political-fact-checking-2/ Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002 Kirman, C. R., Simon, T. W., & Hays, S. M. (2019). Science peer review for the 21st century: Assessing scientific consensus for decision-making while managing conflict of interests, reviewer and process bias. Regulatory Toxicology and Pharmacology, 103, 73–85. https://doi.org/10.1016/j.yrtph.2019.01.003 36 Krimsky, S. (2012). Do financial Conflicts of interest bias research? Science Technology & Human Values, 38(4), 566-587. https://doi.org/10.1177/0162243912456271 La Barbera, D., Roitero, K., & Mizzaro, S. (2023). Combining human intelligence and machine learning for fact-checking: Towards a hybrid human-in-the-loop framework. Intelligenza Artificiale, 17(2), 163–172. https://doi.org/10.3233/ia-230011 Lee, C., Sugimoto, C., Freeman, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64. https://doi.org/10.1002/asi.22784 Le, T. T. (2023). Corporate social responsibility and SMEs’ performance: Mediating role of corporate image, corporate reputation and customer loyalty. International Journal of Emerging Markets, 18(10), 4565–4590. https://doi.org/10.1108/IJOEM-07-2021-1164 Lewandowsky, S., Ecker, U. K. H., Seifert, C. M., Schwarz, N., & Cook, J. (2012). Misinformation and its correction. Psychological Science in the Public Interest, 13(3), 106–131. https://doi.org/10.1177/1529100612451018 Malhotra, N. K. (2006). Questionnaire design. In R. Grover & M. Vriens (Eds.), The Handbook of Marketing Research: Uses, Misuses, and Future Advances (pp. 83–94). Sage. Merpert, A., Furman, M., Anauati, M. V., Zommer, L., & Taylor, I. (2017). Is that even checkable? An experimental study in identifying checkable statements in political discourse. Communication Research Reports, 35(1), 48–57. https://doi.org/10.1080/08824096.2017.1366303 National Geographic (2013). Standards & Practices Handbook. National Geographic Channel. [Internal document]. 37 NC State University. (n.d.). Rubric best practices, examples, and templates. Teaching Resources. https://teaching-resources.delta.ncsu.edu/rubric_best-practices-examples-templates/ RMIT University & ABC. (n.d.). RMIT ABC Fact Check’s Methodology. ABC News. Retrieved March 3, 2025, from https://www.abc.net.au/news/factcheck/about/ Ruiz-Ortega, M. J., Parra-Requena, G., & García-Villaverde, P. M. (2016). Do Territorial Agglomerations Still Provide Competitive Advantages? A Study of Social Capital, Innovation, and Knowledge. International Regional Science Review, 39(3), 259–290. https://doi.org/10.1177/0160017613512652 Sharma, M., Sarin, A., Gupta, P., Sachdeva, S., & Desai, A. (2014). Journal Impact Factor: Its Use, Significance and Limitations. World Journal of Nuclear Medicine, 13(02), 146. https://doi.org/10.4103/1450-1147.139151 Steer, P. J., & Ernst, S. (2021). Peer review - Why, when, and how. International Journal of Cardiology Congenital Heart Disease, 2, 100083. https://doi.org/10.1016/j.ijcchd.2021.100083 Sun, M., Barry Danfa, J., & Teplitskiy, M. (2022). Does double‐blind peer review reduce bias? Evidence from a top computer science conference. Journal of the Association for Information Science and Technology, 73(6), 811–819. https://doi.org/10.1002/asi.24582 Uscinski, J. E., & Butler, R. W. (2013). The Epistemology of Fact-Checking. Critical Review, 25(2), 162–180. https://doi.org/10.1080/08913811.2013.843872 Williamson, M. J., Curnick, D. J., Jacoby, D. M., Durant, S. M., & O’Neill, H. M. (2021). Ethical considerations in natural history film production and the need for industry-wide best practice. Global Ecology and Conservation, 34, e01981. https://doi.org/10.1016/j.gecco.2021.e01981 38 Appendix A: Form Questions Peer Rubric and User Manual Feedback Form Questions • Do you think the rubric covers all relevant aspects needed to fairly evaluate the subject? If not, what should be added or removed? • Were the criteria in the rubric clearly defined and easy to understand? If not, which criteria need more clarity? • How long did it take you to apply the rubric? • Do you think the process was efficient? How could it be streamlined? Any comments about our procedure document can be directed here as well. • Were there any criteria that felt redundant or irrelevant? Were there any important factors missing from the rubric? • Are there any additional comments or suggestions you have for improving the rubric’s efficiency and effectiveness? Filmmaker Questionnaire and User Manual Feedback Form Questions • Were the questions in the questionnaire clear and easy to interpret? If not, which questions need rewording or clarification? • Did the questions focus on the most important aspects of a film and production? Were there any irrelevant or unnecessary questions? • Were the response options (e.g., multiple choice, open-ended) appropriate for each question? Did you feel restricted by any of the options? • How long did it take to complete the questionnaire? Was it too long, too short, or just right? • Was the questionnaire logically organized, with a good flow from one question to the next? If not, how could the structure be improved? • Were there any technical difficulties or confusing elements of the questionnaire? • Do you feel the questionnaire effectively captured the information needed? Are there any questions that should be added or removed? • Did you find the handbook to be a helpful guide? How can we improve this manual to better assist your needs as a filmmaker? • Do you have any suggestions for improving the overall design and efficiency of the questionnaire? 39 Appendix B: Film Experts Contributions Table B1. Name Contribution Carolyn Whittaker Carolyn is a film producer with science background. She provided us with many different contributions, mainly on the background on standards of fact-checking, balanced perspectives, and intellectual properties and their impact on documentaries. Mike Douglas Mike is a professional skier turned filmmakers. He focuses on anecdotes documentaries, which showed us a different perspective of filmmaking. More specifically, he contributed to biases in films, the importance of fact-checking, and the moral component of funding sources. Anny Tubbs Anny is a lawyer turned producer and filmmakers. She contributed in expressing the emotional components of documentaries mixed with scientific evidence, misinformation, and the struggles of funding for smaller films. Appendix C: Adaption of our Tools from Feedback Table C1. Peers feedback Comment and/or suggestions How it was applied to our tools A specific guided approach for performing the review would be valuable. For example, stating to read the questionnaire first, then watch the film while checking questionnaire fact lists, then completing the rubric. This guide has since been added into the handbook Including percentages of editorial power and quality of reference would reduce the vagueness of this scoring. The goal of these criteria was to keep in broad and slightly vague to reduce the onus on filmmakers and allow for educated estimation. Therefore, these categories were not changed; however, in the handbook we added a description of how to estimate these criteria and provided examples of each level. 40 Empty cells in the rubric confused readings with weighing categories. Filled in cells with black to signify an intentional blank space and added a comment into the handbook regarding this. The idea of intellectual property and conflicts of interest was challenging to understand. We added in better definitions, explanations and examples of these terms in the context of films to our handbook. The user manual could be more visually appealing to ease in the reviewing process. Updated handbook with diagrams, flow charts and colouring to guide the process. Table C2. Film industry experts test the filmmaker questionnaire. Comment and/or suggestions How it was applied to our tools Including a rationale as to why we are implementing this screening system. Updated the start of the filmmaker handbook to explain the current challenge of misrepresentation in films and why it is important to ensure films are minimally biases and factually accurate. Including a rationale explaining why we are implementing this screening system. Updated the start of the filmmaker user manual to explain the current challenge of misrepresentation in films and why it is important to ensure films are minimally biases and factually accurate. Filmmakers were only able to attach 1 file to the questionnaire. Updated the questionnaire to allow for multiple file uploads. We should give more context about how filmmakers will be assessed so they can understand the importance of each response. Include the rubric and explain how each film will be scored and evaluated against the passing thresholds. The bias question was challenging to answer as a filmmaker as everyone has inherent biases. Provide a clearer explanation in handbook giving examples of biases and how we will be evaluating inherent and non-impactful biases. Provide an example of how a ‘perfect film’ would fill out the questionnaire. We have not implemented this suggestion as a perfect film is possible. Instead, we have made clearer what the baselines are for success and the failure threshold. 41 Filmmakers were concerned that this level of scrutiny would dissuade filmmakers from submitting their films to BCEFF This is a concern we have discussed with the BCEFF team so we will ‘test’ our framework through a voluntary approach for the 2025 film festival. Appendix D: Final Handbooks D1. Filmmaker Handbook available online via this link or in full below: 42 43 44 45 46 47 48 49 50 51 52 53 54 D2: Film Reviewer Handbook available online via this link or in full below: 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 Appendix E: Step By Step Guide to BCEFF Filmmaker’s Questionnaire 71 Appendix F: Simplified Process of using Fact-Checking and Bias BCEFF Reviewing Tool
Report
eng
Unreviewed
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Undergraduate
University of British Columbia. ENVR_V 400
Developing an advanced selection framework for film festivals
Text