Open Collections

UBC Graduate Research

Enhancing Organizational Capacity for Program Evaluation : The Case of the Neighbourhood Small Grants… Camargo Camargo, Diana Carolina Aug 18, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


310-Camargo_Carolina_Program_Evaluation_Neighborhood_2015.pdf [ 2.17MB ]
JSON: 310-1.0305852.json
JSON-LD: 310-1.0305852-ld.json
RDF/XML (Pretty): 310-1.0305852-rdf.xml
RDF/JSON: 310-1.0305852-rdf.json
Turtle: 310-1.0305852-turtle.txt
N-Triples: 310-1.0305852-rdf-ntriples.txt
Original Record: 310-1.0305852-source.json
Full Text

Full Text

    ENHANCING ORGANIZATIONAL CAPACITY FOR PROGRAM EVALUATION: THE CASE OF THE NEIGHBOURHOOD SMALL GRANTS PROGRAM   by  Diana Carolina Camargo Camargo     A PROJECT REPORT SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  MASTER OF ARTS (PLANNING)  in  THE FACULTY OF GRADUATE STUDIES School of Community and Regional Planning   We accept this project as conforming To the required standard    ………………………………………………. ………………………………………………..   THE UNIVERSITY OF BRITISH COLUMBIA August 2015 ©Diana Carolina Camargo Camargo, 2015  1 Masters Project Report August 18, 2015 Carolina Camargo, MA Candidate UBC | School of Community and Regional Planning Enhancing Organizational Capacity for Program Evaluation: The Case of the Neighbourhood Small Grants Program  1  I feel very grateful for all the support I have received during the elaboration of this report.  The kindness, encouragement, and help of many individuals have contributed enormously to the development of this project.    Vancouver Foundation has been an extraordinary partner. I want to thank specially Lidia Kemeny and Meseret Taye for all the time and effort they have devoted to provide feedback for my work. Lidia and Meseret made sure that I had access to all the resources I needed, and always provided a precise response to my enquiries and requests. It has been a pleasure working with the two of them.  I also want to thank the NSG coordinators who generously participated in the interviews. I greatly appreciate their willingness to support my research and make our encounter possible. It was very inspiring to see their passion for the Neighbourhood Small Grants program and their commitment to its improvement. Thank you so much to all the NSG coordinators for  graciously sharing your time and ideas with me.  My professor Nora Angeles has also been a key supporter in this endeavor. I really appreciate the safe space Professor Angeles created for the expression of my concerns, doubts and struggles.  Thank you for always supporting my dreams and ideas, but also for reminding me of the importance of being strategic and pragmatic. Maraming maraming salamat tita Nora!  Last but not least, I want to thank my mom, family and friends for their continuous emotional support. All of you gave me the reasons and motivation to accomplish this project successfully. Specially, I want to thank my friends who proofread my texts in the last two years. The learnings I acquired through your positive feedback are present in each sentence of this document.  Muchas gracias a todas y todos.  Acknowledgements  2  Table of Contents  I. Executive Summary                     4  II. Introduction                       6 a. About the Neighbourhood Small Grants (NSG) Program  III. Methodology                       9  IV. Program Evaluation: a review                           11 a. Types of Program Evaluation b. A More in Depth View of Impact Evaluation  V. Findings - Understanding the Neighbourhood Small Grants Evaluation Approach                        16 a. About the NSG Evaluation Approach a. 1. Most Useful Components of Evaluation Questionnaires a. 2. Less Useful Components of Evaluation Questionnaires b. NSG Goals  VI. Recommendations                     30 a. General Recommendations b. Recommendations about NSG Goals  VII. Concluding Remarks                    38  References                           39 Appendices                          40    3  List of Figures1  Figure 1. NSG Project 2014 | Bee-coming Leaders: a Bee School for Kids.        4 Figure 2. NSG Project 2014 | My Great Day in Vancouver               10 Figure 3. NSG Project 2014 | Play with Words Crafternoon: Culture Days         17 Figure 4. NSG Project 2014 | Bee-coming Leaders: a Bee School for Kids        18 Figure 5. NSG Project 2014 | Inaugural Main Block Party.              20 Figure 6. NSG Project 2014 | Our Colourful Community. Our Home          23 Figure 7. NSG Project 2014 | DIGS Clean up – West End              25 Figure 8. NSG Project 2014 | Gilpin's 1st Multicultural Potluck Dinner – Burnaby      28                                               1 All the photographs included in this report were taken from the NSG blog: The photographs’ authors are not mentioned on the website. For this reason, photo credits are not included in this report.  4  Program evaluation offers opportunities to build organizational capacity and improve a program’s outcomes. Also, it directs stakeholders’ attention to the most important components of a program. Program evaluation is particularly useful for programs that are continuously adjusted because it provides an evidence base for decision-making. This is the case for Vancouver Foundation’s grassroots grant-making program, Neighbourhood Small Grants (NSG).  Over the years, the evaluation of the NSG program has been very useful for finding “better ways to meet the needs of people, to be relevant, to be accessible. It [evaluation] is a perpetual evolution of trying to do it better”.  Vancouver Foundation, in collaboration with the program’s stakeholders, designed a set of online evaluation surveys that have supplied valuable information over the years. However, the excessive length of these surveys presents a challenge for survey respondents and Vancouver Foundation’s staff. Additionally, a lack of distinction in the dimensions of some NSG goals could affect the evaluation process.  This research aims to answer the primary research question: how can Vancouver Foundation improve the efficiency of the NSG evaluation? and the sub-questions: what is the definition of each one of the NSG goals? What are the purposes of the NSG evaluation?  Executive Summary Figure 1. NSG Project 2014 | Bee-coming Leaders: a Bee School for Kids. – Vancouver West Side  5  A qualitative research design was used to respond to the research questions. The methods employed in this research included the following: semi-structured interviews with Vancouver Foundation’s Staff and NSG coordinators -the local staff of partner organizations in charge of implementing NSG at the local level-; a review of two relevant documents – the book, “Abundant Community by McKnight & Block (2010), and the “NSG Goals and Objectives 2014 Report”. The following are the main findings and recommendations of this research project: Finding: The surveys proved to be very useful for Vancouver Foundation and NSG partner organizations. However, NSG coordinators receive evaluation data that is not specific to their areas. In addition, the current questionnaires are too long. Finding: Open-ended questions better suit the evaluation interests of NSG coordinators. However, these questions require a larger amount of time for data analysis in comparison to closed ended questions.  Finding: The evaluation questionnaire is still too complex for some project leaders.   Finding: The questionnaire does not always distinguish outcomes at different levels and for various stakeholders. Some questions ask project leaders to estimate program outcomes for event participants.  Finding: NSG coordinators are interested in gathering data to improve their work and collecting stories that they can use to attract new participants to the program.   Recommendation 7: Add graphic support to some questions. Recommendation 9: Remove questions that involve making inferences about a third party. Recommendation 8: Distinguish questions about personal, project and community level outcomes. Recommendation 10: Maintain questions that collect feedback about NSG coordinators’ work. Recommendation 11: Maintain questions about the experiences and feelings of NSG participants. Recommendation 1: Identify the purpose of each question, look for similar questions, and find out who will use the information from the question and how. Based on this, decide which questions should remain. Recommendation 2: Alternate annually the focus of evaluation. Recommendation 3:  Share data specific to each NSG community earlier during the granting cycle.  6  Neighbourhood Small Grants (NSG) is Vancouver Foundation’s program that supports residents in the Lower Mainland in strengthening their connections and level of engagement in their neighborhoods. The program offers small grants of up to $1,000 to fund resident-led projects that meet community needs. Since its creation in 1999, the NSG program has significantly expanded its geographic area, number of participants and budget. This expansion has increased the need for monitoring and evaluation mechanisms in order to identify program outcomes and guide future adjustments to the program. During its first decade, the NSG program was subject to various evaluations that were carried out by external consultants. In 2011, Vancouver Foundation decided to develop and implement their own internal monitoring and evaluation system for the program. Since then, the foundation has made four evaluations corresponding to the 2011, 2012, 2013 and 2014 granting cycles. The present document will focus on the evaluation component of the NSG program; specifically, it will look at the evaluation made during the 2014 granting cycle.  There is no unique definition for the term evaluation, partly because it encompasses a very diverse set of approaches and methodologies. In most cases, program evaluation refers to a set of practices that enable decision-making and help to ensure program objectives are met (Gambara & Vargas-Trujillo, 2007). Program evaluation is fundamental for a program’s success because it allows practitioners to reflect on past experiences, assess whether program objectives are being met, and also to adjust the program based on these findings (Estrella, 2000).  According to Vancouver Foundation, the objectives of the NSG program evaluation are to:  Determine if funded projects meet the NSG program goals and targets  Identify the program outcomes from project stories, and project leaders’ and RAC members’ experiences  Understand opportunities and challenges from project leaders, Resident Advisory Committee members and program coordinators To achieve the evaluation objectives, Vancouver Foundation’s staff created a questionnaire for each one of the program’s stakeholders: grantees or project leaders, Resident Advisory Committee (RAC) members, program coordinators and executive directors of NSG partner organizations. These surveys are administered each year between November and December at the end of the granting cycle.  Introduction  7  This professional report is based on an internship with Vancouver Foundation conducted between February and April 2015. This internship involved qualitative and quantitative data analysis of the 2014 NSG evaluation questionnaires, and the writing of a report for Vancouver Foundation. The work carried out during the internship revealed the need to adjust the evaluation questionnaires for two main reasons: 1) the excessive length of these questionnaires presents a challenge for survey respondents and exceeds Vancouver Foundation’s time capacity for data analysis, 2) there seems to be an overlap in some of the NSG goals. An issue that became important as this research progressed was the fact that Vancouver Foundation is redesigning the NSG program. Currently, the organization is exploring the potential of expanding grassroots grant-making from metro Vancouver to the province of British Columbia. The evaluation method and approach will likely change along with the program. For this reason, the findings and recommendations of this report might provide suggestions to support the new model. This report addresses the main question: How can Vancouver Foundation improve the efficiency 2  of the NSG program evaluation? Because a comprehensive evaluation process requires a deep understanding of program goals and evaluation objectives, this project considers two complementary sub-questions: what is the definition of each one of the NSG goals? What are the main purposes of the NSG evaluation?                                              2 Efficiency is defined as the ability to do something with the minimum expenditure of time and effort. This project pursues the following objectives:  Provide suggestions to simplify and streamline the NSG evaluation process  Promote a better alignment of Vancouver Foundation’s needs and priorities with those of community partners  Clarify the definitions of the NSG goals.  About the Neighbourhood Small Grants (NSG) Program Neighbourhood Small Grants (NSG) is one of Vancouver Foundation’s grassroots grant-making programs. It is based on the principles of “Asset-based Community Development” which uses the strengths, skills and expertise of local residents as the building blocks for community development (Kretzmann & McKnight, 1996). Small grants, ranging from mostly $50 to $500, and few grants of $1,000, are given out to support resident-led projects across 11 communities of the Lower Mainland. With the support of Vancouver Foundation and partner organizations, residents work together to design and implement projects that create more vibrancy, connections and engagement in their neighbourhoods. Some examples of supported projects are block parties, community garden enhancements, intercultural cooking classes, and arts and crafts workshops. The NSG program supports community building from the ground up through the use of community members’ skills and knowledge.    8  The Neighbourhood Small Grants program has grown and developed successfully since its creation in 1999. During the first granting cycle, 24 projects were funded and three partner organizations in the city of Vancouver were involved. In 2014, Vancouver Foundation funded 773 projects and coordinated the program in collaboration with 17 partner organizations serving communities in Burnaby, New Westminster, the North Shore, North Surrey, Richmond and Vancouver. In the last five years the number of applications has more than doubled from almost 500 in 2010, to more than 1,200 in 2014. Additionally, the number of funded projects increased from 350 to almost 800 in the same time period. As can be seen in Table 1 the financial resources dedicated to the program have increased as well.   Table 1: Granting Budget between 2012 and 2015 Year Budget 2012 $396,000 2013 $579,000 2014 $619,000 2015 $740,300 This rapid expansion of the NSG program has been accompanied by the need to create criteria and procedures to evaluate the program.  Currently, the NSG program is evaluated using four goals, each of them with their own objectives, outcomes and indicators. The goals are the following 1. Connect and engage neighbourhood residents: build new relationships and strengthen existing relationships among residents, and increase community engagement. 2. Share residents’ skills and knowledge: provide opportunities and spaces for community members to share their skills and knowledge with each other. 3. Build sense of ownership and pride: make use of local resources including people, places and cultures; and promote meaningful contributions by residents to their communities. 4. Respect and celebrate diversity: highlight the diversity within the neighbourhood; connect people across boundaries of age, ethnic backgrounds, income levels, sexual orientations, physical abilities etc.; and respect and promote accessibility and social inclusion.   9  Research for this project was conducted using a qualitative research design. Qualitative research is a field of enquiry that crosscuts multiple disciplines, epistemological traditions and research methods. It consists of a set of practices that interpret and make sense of phenomena in terms of the meanings they have to people (Denzin & Lincoln, 2000). Qualitative research frequently uses a multi-method approach, because it recognises that each practice makes the world visible in a different way. Because objective reality can never truly be captured, this combination of multiple methods, empirical materials and observations in a single study attempts to reach an in-depth understanding of the phenomenon in question (Flick, 2002). The final result is a bricolage, or a pieced together set of representations, about the object of enquiry.  Qualitative research uses an ample set of inquiry strategies such as ethnography, life history, case study and phenomenological techniques. The case study was considered most appropriate for addressing the research questions. A case study is an intensive, holistic, in-depth description and analysis of a single social unit (Merriam, 2014). The case in a case study, refers to a single entity around which there are boundaries; it could be a person, program, institution or community. A case study is suitable for research enquiries where a single unit provides insight about a wider process or issue; or where the purpose is to resolve a problem arising from everyday practice. This reasoning led to the selection of case study as the enquiry strategy for this project; an in-depth understanding of the NSG program would lead to enhancement of the evaluation approach. Additionally, deeper inquiry into the selected case, the NSG program, would likely provide insightful information for the evaluation processes used by Vancouver Foundation and other non-profits.  Methods: This research collected information using the following methods: Semi-structured Interviews: Face to face interviews were carried out with the director and the manager in charge of the NSG program at Vancouver Foundation. These interviews were carried out in order to understand the rationale behind the current evaluation approach, the evaluation purposes for the Foundation, and the program’s goals as defined by the organization (The interview questions are available in Appendix A).  A total of eight interviews (six face to face, one telephonic and one electronic) were conducted with NSG coordinators, staff in charge of managing the program in each community. The purpose of these interviews was to comprehend the coordinators’ understanding of the NSG Methodology  10  goals, as well as uncover their preferences, opinions and needs regarding the evaluation process (The interview questions asked to NSG coordinators are available in Appendix B). Document review: Two relevant works were reviewed as part of this research project. The first was the book “Abundant Community: Awakening the Power of Families and Neighbourhoods” by McKnight and Block (2010). The NSG program and its goals are based on the Asset-based Community Development principles described in this book. The goal of this document review was to formulate a theoretic set of definitions about the program’s goals. Since the goals of the NSG program do not correspond exactly to the concepts formulated in the book, the definitions here included are only approximations to the NSG goals.  The second document reviewed was the “NSG Goals and Objectives 2014 Report”. This report is the document produced for the internship that gave rise to this project. The report was consulted in order to identify the experiences of the project leaders towards achieving each NSG goal. Figure 2. NSG Project 2014 | My Great Day in Vancouver – Mount Pleasant, Little Mountain  11  This section provides a brief review of the most important aspects in the field of program evaluation. It starts by defining program evaluation and the most common uses of evaluation results. The segment continues with a description of the multiple methods for, and theoretical conceptions about program evaluation. Subsequently, the multiple types of evaluation are briefly reviewed. Since the process that concerns the present research is an impact evaluation, this section provides a more in depth overview of impact evaluation.  Program evaluation is a common activity in many academic and professional fields. It is widely used by governments, educational institutions, non-profit organizations, development agencies and private industries. It usually occurs in conditions that involve multiple and conflicting stakeholders and where resources and time are limited (Trochim, 1998). Most theorists define program evaluation as a practice that uses formal methodologies to collect and synthesize information about the quality, value and outcomes of a social program (Mertens & Wilson, 2012). The information produced as a result of a program evaluation tends to be used to:  Make decisions about the future of the program (Lewis, 1994)  Correct mistakes and improve program implementation (Lewis, 1994)  Assess whether project objectives have been met (McDavid, J. & Hawthorn, 2006)  Ensure accountability for spending of financial resources (Estrella et al., 2000).  Make a value judgement about program results (Lewis, 1994)  Inform policy making (Estrella et al., 2000).    Program evaluation uses a diverse set of research methods that include both qualitative and quantitative approaches. The methods most commonly used are surveys, interviews, focus groups, observations and ethnographies. Since qualitative and quantitative methods provide different types of information, it is desirable to include both approaches in a program evaluation (Adato, 2012). On one hand, quantitative methods provide uniform measures to assess whether or not a program has generated changes. They also allow comparison of program outcomes across time, geographic regions, program phases and so forth. On the other hand, qualitative methods The particular characteristics, methodologies and purposes of program evaluations vary depending on the context where the evaluation is being done. Program Evaluation: a review  12  provide insights and information explaining why implementation of a program has or has not produced changes; qualitative methods are “crucial to understanding impact, as opposed to simply measuring it” (Rao and Woolcock 2003). They also allow for the improvement of survey design and for the identification of unanticipated impacts and unquantifiable outcomes. A mixed-methods evaluation design, that includes both quantitative and qualitative methods, offers a more coherent, reliable and useful information set than an evaluation design that only includes one type of method.  Program evaluation is understood differently by various theorist; therefore it has been necessary to create typologies to explain the diverging conceptions about this activity. According to Guba and Lincoln (1989), the first and second generations of evaluation focused on measuring progress and describing the program’s strengths and weaknesses. These models conceive of evaluation as a method for “providing the most accurate information practically possible in an even-handed manner”; additionally, the evaluation process should not include any judgement about the relative success or failure of a program because evaluative enquiry should remain removed from politics (Berk & Rossi, 1999). In contrast, the third generation of evaluation takes value judgement as its defining characteristic (Guba & Lincoln, 1989). From this perspective, program evaluation is intrinsically linked to political power and decision making about societal priorities and paths of action (Greene, 2000). Evaluation encompasses both an empirical aspect (the information collected) and a normative aspect (a judgement about the object of evaluation) (Mertens & Wilson, 2012). Therefore, the values and standards that will guide the evaluation must be established from the beginning. These standards usually come from needs assessment results, program objectives, program manager preferences or priorities of program participants (Mertens & Wilson, 2012).  Starting out from the stance that program evaluation happens in an inherently political context where power is exerted and decisions are made, the fourth generation of evaluation goes even further. It centers in participation and the incorporation of all stakeholders in the evaluation process, which traditionally was oriented exclusively to the needs of funding agencies and decision-makers (Guba & Lincoln, 1989). When doing participatory evaluation, the main questions are, who is benefiting and learning from the evaluation process? Whose concerns, priorities and definitions of success are guiding the evaluation? Who counts reality? And whose reality counts? (Estrella & Gaventa, 1998). In this fourth generation, evaluation is viewed as an intervention that enhances the program’s outcomes and increases organizational capacity (Horton et al., 2003). Those involved in the participatory evaluation process often experience changes in the way they think and act as a result of this process (Estrella et al., 2000). Participating in the creation and implementation of an evaluation mechanism also develops a stronger sense of ownership, which in turn strengthens the basis for learning. Since its design, evaluation raises questions that have an immediate impact in program implementation (Horton et al., 2003). For  13  example, by requiring the clarification of the program’s values and goals, evaluation directs the stakeholder’s attention to the core components of the program. This attention to priorities can be maintained during program implementation through the selection of relevant indicators. An expression that summarizes this approach is “what gets measured, or evaluated, gets done”. Well-designed evaluation reinforces what the program is trying to do and provides a learning opportunity for those involved in the evaluation process. Despite the apparent contrast between the different generations of evaluation, in reality the distinctions are not as straightforward. In most evaluation processes some features could be classified as a particular generation of evaluation and others could be assigned to a different one. Regardless, it is important to have an understanding of the multiple ways in which an evaluation process can be conceived.  Types of program evaluation  Ideally, program evaluation should be carried out at all stages of program development. As can be seen below, at each state the evaluation’s goals and objectives are different: Proactive Evaluation: The goal of this type of evaluation is to collect information that will help to make better decisions about the future of the program.   Needs assessment: occurs during the stage of program design and identifies what is needed to transform the situation that is being addressed by the program. A needs assessment allows for understanding of the problem, or topic of intervention, and also to establish a base line for future evaluations.  Design evaluation: occurs once a tentative program design is selected. It consists of a detailed analysis of the potential strategies and activities that constitute the program. This analysis provides information decide if the organization has all the resources required to implement the program and if the tentative strategy is well suited to reach the program’s objectives.   Process evaluation/monitoring: documents the program’s activities and contrasts them with the initial program design. This type of evaluation is made to assess if the program is being implemented in the way it was planned, and to identify whether adjustments need to be made in case there is a deviation from the initial plans.  Retroactive evaluation: This type of evaluation aims to give an account of program execution and justify spending associated with the program.  Impact Evaluation: this type of evaluation aims to determine the short, medium and long term effects of the program. Although impact evaluation can be carried out during program implementation, it is regularly made after the implementation cycle has finished. Table 2 summarizes the different types of evaluation:   14  A More in Depth View of Impact Evaluation Impact evaluations generally have a summative role, which means they report on the achievements of the program. When doing an impact evaluation, it is important to carry out certain activities to ensure that the evaluation process is thorough enough for the desired level of results; this series of recommended steps is called logic of evaluation. A logic of evaluation model followed by many evaluation authors (Fournier, 1995; Rogers & Owen, 2001) encompasses four activities:  1) Establishing criteria, this activity looks for an answer to the question “On what dimensions must the subject being evaluated do well?” This initial activity is very important because a comprehensive evaluation can only be made when the principles that guide the intervention are clear. The following are the criteria most commonly used to set evaluation criteria:  Objectives: The worth of a program is judged by analysing the extent to which the objectives of the program have been achieved. In this Table 2: Types of evaluation Table 2: Types of evaluation What is being evaluated? Purpose of the Evaluation Who leads? With who? How? Needs Assessment: Strategies necessary to transform the situation Proactive Evaluation: To make decisions regarding the  future of the program Internal Evaluation: People from within the organization or program participants Non-participatory evaluation: Evaluation team defines process without involving stakeholders Quantitative Evaluation: Process produces quantifiable numeric data Design Evaluation Potential strategies for program Retroactive Evaluation: To account for program spending and activities External Evaluation: People or organizations that are not responsible or linked to program Participatory Evaluation: Process is defined and implemented by all stakeholders Qualitative Evaluation: Process produces non-quantifiable information about meanings, feelings motivations Monitoring/ Process Evaluation: Program implementation  Mixed Evaluation: People involved in the program and external evaluators  Mixed-methods Evaluation: Produces both quantifiable and non-quantifiable information Impact Evaluation: Program results      Source: Rogers & Owen (2001), Berk & Rossi (1999), McDavid, J. & Hawthorn (2006)  15  case, program objectives are used to organize data and define the evaluation criteria.  Participants’ Needs: In this case the impact evaluation measures the extent to which the program meets the needs of the participants. Needs-based impact evaluation is recommended in cases where participants have not been part of defining program objectives because it is possible that the established objectives do not represent the needs of the participants.  Program Results: This involves determining the unintended outcomes of the program without regard to the program’s objectives. Its purpose is to identify both positive and negative changes that were generated as a result of the implementation of the program.  Theory: This impact evaluation requires that evaluators define the theory underlying the program, as well as the mechanisms that relate program actions and outcomes. This type of evaluation allows for explanation of why certain actions may lead to particular results. 2) Constructing standards, this activity aims to answer how well the object being evaluated should perform. This activity involves the creation of measurable and specific goals and indicators that will allow for measurement of the program’s progress.  3) Measuring performance and comparing with standards, will help to answer how well the object being evaluated performed. 4) Synthesising and integrating evidence into a judgment of worth, this final activity asks the question “What is the worth of the object being evaluated?” In this case, the evaluator makes a judgement about the impact of the program based on the criteria and standards previously selected.    16  Understanding the Neighbourhood Small Grants Evaluation Approach  This section summarizes the main research findings of the present study. The first part reviews the NSG coordinators’ assessment of the evaluation approach and questionnaires. The second part compares the NSG program’s goals with the defining characteristics of an abundant community (McKnight & Block, 2010).  Note to the reader: The sources used in this research give multiple definitions of the NSG goals. None of these definitions is seen as more appropriate than the others. Similarities are pointed out with the aim of identifying areas of coherence and agreement across sources. These areas can indicate paths to move forward with the NSG program. Divergences are also identified in order to indicate segments where the program can be defined more precisely or where changes should be introduced.  About the NSG Evaluation Approach NSG coordinators expressed overall satisfaction with the NSG evaluation approach. They consider that the surveys and the information they provide are highly valuable for project leaders, RAC members, Vancouver Foundation and themselves. NSG coordinators reported that, in addition to Vancouver Foundation’s annual report, they would also like to receive the evaluation data specific to their community. Coordinators mentioned the importance of receiving the evaluation results earlier in the year; this way they could include these findings in the reports some of them produce for partner organizations.  Currently, NSG coordinators use informal mechanisms to evaluate the program. During orientation sessions, end of the year celebrations and NSG events, coordinators chat with project leaders and RAC members about things that need to be modified or adjusted in the program. Only one NSG coordinator is carrying out their own evaluation process in addition to the one made by Vancouver Foundation. The remaining coordinators consider that additional evaluation would be unnecessary for them and excessive for RAC members and project leaders. Coordinators use the evaluation results for the following purposes:  To identify areas that should be maintained and areas that need improvement.  To identify geographic areas and socio-demographic groups that require additional promotion.  To account for the program’s results to partner organizations and RAC members.  To plan for the following year. Findings  17  In addition, NSG coordinators think that the evaluation helps Vancouver Foundation to track whether they are meeting their own goals for the NSG program, and gives an opportunity to project leaders to reflect back on their project.  The combination of open-ended and closed-ended questions provides multiple types of information that suit the diverse preferences and needs of NSG coordinators. While most coordinators expressed a preference for open-ended questions, some of them showed a stronger interest in the concise and comparable data obtained from closed-ended questions. Despite the survey’s usefulness, the length of the questionnaires posed a dilemma for many NSG coordinators; as it was explained by a coordinator:  “It is a good evaluation and the data that they collect. I like it. I think it is a bit long but if you want to collect useful data I don’t think you can have a super short survey. So that is something we always struggle with.” Some coordinators mentioned that shortening the questionnaire could impact the quality of the evaluation results while others suggested that it would be worth reducing its size. NSG coordinators emphasized the importance of making the evaluation questionnaires as simple as possible. They consider the current questionnaires very clear and acknowledge Vancouver Foundation’s efforts to use plain language and simple question structures. However, NSG coordinators pointed out that the electronic and written format of the survey still poses challenges for people with low literacy levels or whose mother tongue is not English.  “I think it is good and easy. At least for the population I am dealing with here, it doesn’t seem to be a difficult task for them. The questions seem to be easy for them to answer. The Survey Monkey works great for them” “I would also suggest offering to do this verbally somehow, but I’m not sure how that would happen. As it is, this seems to require a high level of literacy and comfort with writing.” NSG coordinators identified a problematic characteristic of the Project Leaders Survey. In this survey most questions ask about project outcomes (e.g. Did your project involve an expert teaching participants?), and some ask about project leaders’ experiences (e.g. What inspired Figure 3. NSG Project 2014 | Play with Words Crafternoon: Culture Days – North Vancouver  18  you to lead a project?). However, a few questions, mainly in the objectives section, ask about NSG participants’ experiences (e.g. would the people who connected through your project likely see each other again? did your project help participants to learn from each other?). This last type of questions is not desirable because it requires a high degree of inference from project leaders, which means the responses generated are subjective. Additionally, in terms of data analysis, is not desirable to merge program outcomes at a personal level (e.g. has your involvement in the NSG program increased your participation…?), an interpersonal level (e.g. Did your project connect people from different backgrounds…?) and a community level (e.g. Do you think your project enhanced the sense of ownership and pride within the local community?). Although is important to identify the impacts the program is having at each level, it is also essential distinguish the outcomes that are occurring at each level.  Most Useful Components of Evaluation Questionnaires  When asked about particular sections of the survey, NSG coordinators identified the ones that offered the most useful information for them (The questionnaire for project leaders can be consulted in Appendix C and the questionnaire for RAC members can be observed in Appendix D; Appendix E presents the comments of NSG coordinators about the project leaders questionnaire). According to NSG coordinators, the most important sections of the project leaders’ questionnaire are I. Project description, III. Program support and personal impact. Regarding the RAC members’ questionnaire the III. Program support, and VI. Personal impact and capacity sections were featured as more informative. NSG coordinators showed favoritism for open-ended questions that promoted storytelling about NSG projects. The questions about what worked well (PL #7, RAC #10), what were the challenges (PL #8, RAC #11), what inspired them to lead a project (PL #10), memorable experiences (PL #31, RAC #17), and how their participation in the program changed Figure 4. NSG Project 2014 | Bee-coming Leaders: a Bee School for Kids. – Vancouver West Side  19  their relationship to the community (PL #30, RAC #24) were identified as the most useful. Coordinators are very interested in gathering information from project leaders and RAC members about their feelings towards the NSG program. They believe the questions listed above provide information they can use to engage more residents and to evaluate the impact of the program. “I like the questions, what worked well and what were the challenges. What inspired you to lead a project? Maybe I can recreate that inspiration for someone else, so that they will join. What other support will be helpful to you. There is a lot of questions about how well the project went and that is important to know but for me I am more interested in the questions of how did they become engaged, how can I keep engaging them, how can I better engage them.” “I think it’s good to have some of the foundational info around how the goals and objectives were received and guidelines followed, but I think the most important thing is to capture the SPIRIT of the grants – what magic happened? How did it make people feel? What connections happened? How did it create new leaders?” NSG coordinators also rated as useful the set of questions that provide feedback about the support offered to participants (PL #12, RAC #18) and additional types of support that would be helpful for them (PL #14, RAC #19). These questions are important because they allow participants to express openly their opinions about the coordinator’s work and also provide information to guide adjustments for the coming years.  “Question 12 is very useful because it provides feedback about the activities undertaken by NSG partners. And 14 as well because it suggests what else can I do to support project leaders.”   Less Useful Components of Evaluation Questionnaires In general, NSG coordinators regarded the evaluation questionnaires highly; some of them even remarked that the entire questionnaire was very useful. Other NSG coordinators mentioned that some of the survey’s sections and questions did not provide new information, and, therefore, were less useful for them. This was the case for the project leaders’ questions about the project’s location (PL #5) and description (PL #6), and types of local support received (PL #11). Regarding the RAC members questionnaire, some NSG coordinators reported that they already know most of the information because they work closely with RAC members and get their feedback frequently. NSG coordinators acknowledged that although some questions don’t provide relevant data for them, these data are useful for Vancouver Foundation.  “For the RAC questionnaire information I don’t really use it because I know most of the information already; I communicate with them directly and ask them for feedback.”    20  Connecting and engaging neighbourhood residents is considered the main goal of the NSG program by Vancouver Foundation staff and NSG coordinators. It is perceived as something that is lacking in neighbourhoods, and also as the starting point for reaching the other goals of the program. “This is the heart of the program and what we are trying to achieve. More than the projects themselves, the impacts on people and their I. CONNECT AND ENGAGE NEIGHBOURHOOD RESIDENTS   …………………………………………………………………………… Over 96 people attended the event.  The highlight of the evening was a game where volunteers worked together to overcome a presented challenge. The scene of this diverse group of people standing on a small platform and working together was an impactful metaphor of how powerful communities can be when they work together to solve problems. This can only be reached when people know and trust each other. The block party has made a lasting impression and hopefully has inspired others to start a similar initiative in their own communities.”  ……………………………………………………………………………  Figure 5. NSG Project 2014 | Inaugural Main Block Party: Connect. Community. Celebrate. – South Vancouver NSG Goals  21  feelings of connection and belonging in the neighbourhood is what we feel is the most important goal of the program.” Currently, some coordinators are focusing more on the connection component of the goal while others are more dedicated to strengthening the engagement piece. This means that the criteria used by NSG coordinators to measure attainment of this goal, although related, vary between communities.   “We are working so that residents have the opportunity to meet other people and connect. The engagement part might come in the future, but now we are working on the piece around bringing them together and out of isolation.” The “connect and engage neighbourhood residents” goal was defined in the following ways: According to the Abundant Community Model: A community becomes powerful and competent under “The Presence of Association”. This means that a community is stronger when its members come together voluntarily to do something out of a common interest. Community associations are repeatable; have continuity; emphasize actions that strengthen the social relationships in the community; and their members are connected by choice, affection and reciprocity. Associations are independent of formal institutions, though they can be supported by them. According to Vancouver Foundation’s Staff: This goal has two components that build upon each other. The first one, connecting residents, involves “creating new relationships between people and deepening existing relationships”. This component is about people interacting, and learning to be with each other and to work together.  The second component, engaging residents, goes further and involves a more active role. To engage means to participate in things that are happening in the community such as clubs and organizations or to participate in the creation of new collective spaces and opportunities. This engagement in the neighbourhood is expected to remain beyond the life of the NSG project where residents are involved.  According to NSG Coordinators: Neighbourhoods are stronger when people have relationships and know each other. Specifically, people in a neighbourhood connect once they meet or get in touch with one another. To be connected with neighbours is to become familiar with them, to know who they are and where they live. Engagement is long-term involvement in a neighbourhood that requires ongoing participation and the development of relationships in the community. Engaged residents work together, know their neighbours, and join existing initiatives offered by local organizations, institutions and governments. Comparing definitions: The definitions given by the Abundant Community model, Vancouver Foundation’s staff and NSG coordinators all emphasize that community participation and social relationships are at the core of connecting and engaging neighbourhood residents. However, meeting neighbours and  22  creating new relations among residents, which are part of this NSG goal, are not explicit components of the model proposed by McKnight and Block. The reason for this extra component is that many residents in the Lower Mainland are isolated from their neighbours; therefore, in the context of the NSG program, connecting residents became a precondition for the creation of association or engagement. As explained by Vancouver Foundation’s staff: “Initially the purpose of the program was to bring residents together working collaboratively to improve their neighbourhoods. However, Vancouver Foundation found that people didn’t know each other, so connecting people became one of the program’s priorities. People need to know each other before they can work together.” Deepening existing relations, another aspect of connection and engagement as described by Vancouver Foundation’s staff and NSG coordinators, was defined ambiguously. While Vancouver Foundation’s staff included this characteristic as part of the connecting component of the goal, some NSG coordinators saw the deepening of relations as a sign of engagement. This means that for some NSG coordinators engagement involved strictly active participation in community activities, whereas for others it also included having lasting relationships in the neighbourhood.    23  The definition of this goal is very clear, almost self-explanatory. As is shown below, there is a high level of coherence in the way this goal is explained by McKnight and Block (2010), Vancouver Foundation’s Staff and NSG coordinators. This goal is perceived as very empowering for local residents, and as one of the keys for keeping people engaged in their neighbourhoods.   “I think that sometimes the money that we are giving out is not so important. It is the permission that we are giving to people to say “yeah, you have something to share, you have skills and knowledge that other people will value” and here we want to affirm that with this little bit of money. And we see people really blossom and offering what they have to offer with that.” II. SHARE RESIDENT’S SKILLS AND KNOWLEDGE  …………………………………………………………………………… “Our guest instructor led us in a brief introduction to acrylic painting. It was a big help to the beginners and a nice reminder for the more experienced. Now the room was filled with artistic energy! You could feel a rainbow of colours floating in the air! It felt really good to have a group of people painting together with you in a relaxed and non-judgmental environment. “I can’t wait to see and attend more workshops of this kind in our neighbourhood in the future!” …………………………………………………………………………… Figure 6. NSG Project 2014 | Our Colourful Community. Our Home – Cedar Cottage, Hastings, Renfrew.  24  According to the Abundant Community Model: called the “Giving of Gifts” by McKnight and Block, this principle states that neighbourhoods already have all the skills and resources they need. Every resident has something precious to offer (a gift) and most of the time they are eager to share their capacities with their neighbours. A competent community focuses on the gifts (skills and knowledge) that are present in the neighbourhood by identifying what each person does well and getting them to do it. When individuals’ gifts are combined, they create new possibilities and their power is multiplied. A community that uses its gifts is more resilient because it can provide for itself most of what would have to be purchased otherwise.  According to Vancouver Foundation: Residents often forget that they have the tools to solve their problems because they have placed an excessive reliance on resources from outside their community. This goal focuses on making use of unrecognized local talents and skills, and it is manifested in two ways. The first is to share a specific set of skills or knowledge capabilities with other local residents through workshops or classes. The second is to create opportunities and spaces where all residents work together using their unique skills and knowledge.  According NSG coordinators: Every neighbourhood has myriad capacities and resources, and every resident has something valuable to contribute. However, people forget to look around them and quickly turn to the market for the resources they need or desire. The Neighbourhood Small Grants program allows residents to recognize their own skills and their neighbours’ skills. This way they find out that they can support each other with those skills. The program also provides opportunities for transferring skills through mentorships and workshops; residents build their skills and knowledge by attending to these types of NSG projects. Project leaders and those who collaborate also build their skills through the preparation and organization of the projects.  Comparing definitions: All the definitions offered emphasize that every community member has valuable skills and knowledge to offer, and neighbourhoods are stronger when these skills are recognized, shared, and valued. Neighbourhoods can satisfy most of their needs by using the skills and knowledge that already exist within them. In general, the three definitions were very similar. The only element that was found in only one definition was developing individuals’ skills and knowledge. The main aspect of this goal is to discover and use the skills each person has; to learn and acquire skills and knowledge seem to be secondary, if important at all. Some NSG coordinators mentioned that currently the NSG program does not always fully articulate that everyone has something to offer. This decreased emphasis could explain why, when defining this goal, some NSG coordinators stressed learning new skills over recognizing and using existing skills.     25   This goal is considered to be closely related to the other NSG goals, especially “Connect and engage neighbourhood residents” because connecting with other community residents creates a sense of ownership. Ownership and pride are seen as required attributes that enable people to create strong communities. This goal did not have a corresponding concept in McKnight’s and Block’s book. The definition below was built taking excerpts of the book that addressed similar ideas to those provided in the interviews.  III. BUILD SENSE OF OWNERSHIP AND PRIDE WITHIN COMMUNITIES  …………………………………………………………………………… “Over the last two years, we have also started taking care of the common areas, and with a number of grants this year we were able to make the East Deck quite spectacular. My favorite sight is the bird bath and two whiskey barrels full of stunning plants and blossoms (see photo). Patients and visitors alike enjoy this area immensely.”  …………………………………………………………………………… Figure 7. NSG Project 2014 | DIGS Clean up – West End  26  According to the Abundant Community Model: People do things voluntarily for their community because they care about it (ownership). Furthermore, people’s contribution to their community increases their sense of worth (personal pride). A neighbourhood grows when its residents contribute and exchange goods and services locally.  According to Vancouver Foundation: To build a sense of ownership is to strengthen residents’ feelings of belonging to their neighbourhood. Ownership is expressed when people take responsibility for each other, the physical environment in their community, and the challenges that arise in the neighbourhood. A community that has ownership finds solutions for things that are going wrong in the neighbourhood. This goal is also about getting to know local institutions, business, and residents and feeling familiar within the community. To have pride in one’s neighbourhood is to feel a sense of satisfaction with it. It is to feel good about one’s community, block and neighbours. This feeling is developed when people make meaningful contributions to their communities. Ownership gives a sense of pride.  According to NSG Coordinators: This goal is about developing and sharing resources, being part of activities and traditions, and celebrating “uniquenesses” in the neighbourhood. Ownership and pride are about showcasing the neighbourhood and the things that are happening in it. They also involve creating a sense of place. All this is more easily perceived when there is a tangible result that everyone can see, for example, clean-up, place making or art installation projects.  NSG coordinators mentioned that ownership and pride are different but intertwined. Specifically, ownership is a feeling that the place where people live is their community and they belong to it. In order to have ownership, people need to feel a sense of agency and control over their neighbourhood; they need to feel they can make a difference. When people have ownership they are active agents of change, they do things to make a better community and they take responsibility for both the good things and the bad things that happen in the neighbourhood.  Pride is feeling good about the area where one lives. It occurs when people do something for their neighbourhood and they feel happy about what they have accomplished. Another source of pride is when people know their neighbours. A sense of pride is built when residents get together to improve the physical look of their neighbourhood. People cannot feel proud of their neighbourhood until they have ownership of it. Comparing definitions: The definitions compiled from Vancouver Foundation’s Staff, NSG coordinators and the book “Abundant Community” all indicate that people have ownership when they feel responsible for their neighbourhood, and they take action to improve it. People have pride when they feel good about the things they have accomplished working for their neighbourhood, and when they feel good about the place where they live. In the NSG context, building ownership and pride  27  is associated with place making and physical improvement projects.  As can be seen in the three sets of definitions, a broad range of attributes was used to describe the “Build sense of ownership and pride” goal. The use of local resources and institutions, which is a central aspect of the goal according to Vancouver Foundation, was not mentioned by any of the NSG coordinators. Furthermore, elements that characterize other NSG goals such as getting to know one’s neighbours (Connect and Engage), participating in local traditions (former goal, Creating Lasting Impact) and celebrating uniqueness (Respect and Celebrate Diversity) were included as constituting parts of this goal. The former findings show that some components of the goal lack a clear and shared definition. To complicate things further, the term ownership poses a difficulty for this goal. If having ownership is doing something for one’s community, all NSG project leaders have ownership by their mere participation in the program. This issue was identified for some coordinators: “I think that all of our projects do this. I think that just by getting together with your neighbours and doing something there is a sense of ownership and pride.” Therefore the ownership piece of this goal is not informative, nor does it provide any feedback about the performance of the program.     28   According to the Abundant Community Model: Every person is unique. A competent community knows this and encourages all its members to be themselves. Community members sometimes need to be asked to join and contribute, yet they will express their full potential only when this invitation is extended with the “compassion of hospitality” or in other words in a welcoming and accepting context. This is especially the case for people that are usually left out of community life, or people who are differentiated by society because of their conditions (e.g physically disabled, youth-at-risk, immigrant, low-income).  According to Vancouver Foundation: people often have strong ties within their own social groups but they rarely have opportunities to interact with other groups This goal is about bringing together people from different walks of life such as different age groups, cultures, religions, sexual orientations and socioeconomic levels. When diverse people interact with each other in a friendly environment, they build competence to accept different ways of doing and different ways of being. In turn, these competences allow them to be together in a way that is rewarding for everyone. Respecting and celebrating diversity also includes creating IV. RESPECT AND CELEBRATE DIVERSITY  ………………………………………………………………………….… We literally ate around the world from countries such as Japan, Korea, China, India, South Africa, Portugal, and more! There were multicultural arts and crafts such as origami and flag making. In addition, there was multicultural performances: of Irish fiddle and dance; plus, children singing songs from Japan, China, and Mexico. It was an amazing evening that brought a community closer while learning more about each other’s culture and heritage.”  ………………………………………………………………………….…    Figure 8. NSG Project 2014 | Gilpin's 1st Multicultural Potluck Dinner – Burnaby.  29  spaces for people with shared life experiences, so they can come together and build their own support and social networks.  According NSG coordinators: A community respects and celebrates diversity when everyone is included and welcomed (e.g. people with diverse sexual orientations, ages, physical abilities, cultures, and income levels). In the Lower Mainland, diversity is very common because there is a wide range of people living here. Although this continuous interaction can bring understanding about differences, this is not always the case. People have different levels of barriers to participating. Therefore, diverse environments might not be safe spaces for everyone. To truly celebrate and respect diversity, it is necessary to take into consideration marginalized groups and what they require to feel included. This goal also involves supporting the creation of both safe spaces that offer support for diverse communities, and opportunities for interaction among diverse communities.  Comparing definitions: In general, the three definitions were very similar and stressed that people in a neighbourhood celebrate and respect diversity when they invite, welcome and include all the members of their community; especially those with diverse sexual orientations, ages, physical abilities, cultures, religions and socioeconomic levels. Vancouver Foundation and NSG coordinators indicated that the establishment of inclusive spaces for everyone should be complemented by the creation of safe spaces for people with shared life experiences. Some NSG coordinators went further and stated that this goal should address social inequalities and explicitly promote the adoption of measures to remove the barriers and access gaps that some people face. “Diversity also needs to have an understanding of marginalized groups. Just because you have a diversity maybe by accident, that doesn’t necessarily mean that there is a safe space for them. There has to be an understanding of marginalized groups and how to be inclusive. So, if you achieve diversity by accident that doesn’t necessarily count.”      30  This section offers a series of recommendations based on the research findings. The first part of this section offers a series of general recommendations for question design. The second part, gives recommendations about ways to clarify the NSG goals, shows activities at a neighbourhood level that could serve as indicators for the achievement of goals, and lists items recommended by NSG coordinators through the interviewing process. Since this project is developed by an external party, this report does not provide a specific judgement about the evaluation questionnaires. It leaves it to the program’s stakeholders to decide how well the current questionnaires fulfill their needs. It is expected that this information serves as a guide for the changes that the NSG program is undertaking.   General Recommendations  1. Review each question item in the questionnaire: It is important to identify the kind of information that each question is providing and how useful it is. It is also important to look for other questions that might give similar, or the same, information. If that is the case, review to understand if it is necessary to have multiple questions that address the same concept or program dimension for triangulation purposes. Some guiding questions for this reflection can be: what is the purpose of asking this question? What information will it provide? Who would use this information and how? This process will help to identify questions that can be removed from current questionnaires.   2. Alternate evaluation focus: This recommendation came from one NSG coordinator. The person suggested focussing the evaluation on a different aspect or topic every year; for example a different theme or goal. The evaluation could assess all aspects every year but could do an in-depth inquiry for a particular topic. This would allow for comparable data across time, and would provide the flexibility needed to do a more comprehensive evaluation of relevant aspects. Alternating the focus of evaluation each year could offer more of an incentive to participate; especially for RAC members and project leaders who have been involved in the NSG program for several years.   3. Share evaluation data specific to each community and earlier during the granting cycle: NSG coordinators could make a better use of the evaluation data if they can get data Recommendations  31  specific to the communities in which they work and if they could access the data earlier.  4. Maintain questions that provide feedback to NSG coordinators regarding their work and ways to improve it: NSG coordinators are interested in identifying ways of improving and facilitating the experience for project leaders and RAC members. Some questions that address this topic are: Project Leaders Survey: 7, 8, 12, 13, 14; RAC Members Survey:8, 10, 11, 14, 15, 16, 18, 19, 20, 25)  5. Maintain questions about the experiences and feelings of RAC members and project leaders: NSG coordinators want to expand the NSG program in their neighbourhoods and reach out to new people. Therefore, they want to have emotionally appealing information regarding the impact the NSG program has on people’s lives. Some questions that could be useful for this are: Project Leaders Survey: 10, 18, 30, 31; RAC Members Survey: 12, 17, 24).  6. Introduce rating a scale for some questions: Rating scale questions are useful to measure gradation in opinions, attitudes and behaviours. They are also popular for measuring the magnitude of change perceived by respondents. NSG coordinators suggested replacing some yes/no questions with scale questions because the latter are more informative. Some of the suggested questions are: Project Leaders Survey: 12, 19; RAC Members Survey:  15, 16, 18, 21d, 22, 23, 24. To ensure better interpretation by respondents, rating scale questions should have between four and six points, and provide a label over each point (Fowler, 1995). The following are examples of scales that could be used:  Not at all useful Not Very useful Very useful Extremely useful  Very dissatisfied Dissatisfied Satisfied Very satisfied  7. Adding graphic support to some questions: coordinators expressed that an exclusively written questionnaire creates a barrier for NSG participants who do not feel comfortable expressing themselves by writing in English. The inclusion of visual aids to accompany written instructions can be used as a strategy to address this issue. Some symbols that could be added are:              8. Include more closed-ended questions: According to De Vaus (1991) closed-ended questions are very useful when a questionnaire is long, because they are quick to answer. To be exhaustive when developing response options, data from qualitative questions should inform this process. The NSG questionnaire would benefit from including more closed-Thumbs up (good) and down (poor) Faces (poor, good, average) Arrows (increase/decrease)  32  ended questions; especially in cases where open-ended questions result in a limited range of responses. For example, in this open-ended question of the Project Leaders Survey “Did your project involve an expert teaching participants? What skills or knowledge were taught?” only four types of skills (i.e. arts, gardening, cooking and organizing) were mentioned by more than 20 participants. A modified question could look like this:  20. Were any of the following skills or knowledge taught by an expert? (Please check all that apply) a. Arts b. Gardening c. Cooking d. Organizing e. Other  ___________________  9. Reduce the number of open-ended questions: The analysis of open-ended questions demands a considerable amount of time (De Vaus, 1991). For this reason, it is desirable to decrease the amount of open-ended questions in the NSG surveys. However, since the program’s stakeholders are very interested in the storytelling and emotive components of the NSG program, some open-ended questions should be maintained. Open-ended questions should be maintained when it is important to gather answers in the participant’s words or when the concept in question is hard to measure in numbers. Since they were almost unanimously highlighted by NSG coordinators, it is recommended to maintain the following questions:  What worked well with your NSG project? (Project Leaders)  What were the challenges of your NSG project? (Project Leaders)  What inspired you to lead a project? (Project Leaders)  What is your most memorable experience as a RAC member (RAC members)  10. Avoid questions that involve making inferences about the behaviours, emotions or experiences of a third party: Remove or adjust the questions that ask project leaders about the personal experiences of participants in their NSG events. These questions are: 18, 23, 29.  11. Distinguish between questions about personal (project leaders, RAC), interpersonal (project participants, and community (neighbourhood) level outcomes: create separate objectives, outcomes, indicators, and evaluation questions for the expected impacts at each level.    Characteristics of Good Interview and Survey Questions  1. The question means the same thing to every respondent.  2. Use simple language and define complex terms if they are used.  3. Ask for only one concept or aspect of a concept. If the evaluator wants to ask for different aspects or concepts, they should create a separate question for each concept.  4. Don’t lead the respondent towards a particular answer choice  5. Is short and direct to avoid ambiguity  Sources: Fowler (2009); Weisenberg, Krosnick, & Bowen, (1989)  33  Recommendations about NSG Goals  1. Create multiple questions that measure this goal: Since this is unanimously considered by staff as the most important goal of the NSG program, it is worth maintaining multiple measures of its attainment. Multiple questions would help to distinguish between the levels of attainment of each one of the goal’s components: connection and engagement.  2. Create questions that provide a more direct indication of the goals attainment: Questions 15 and 16 enquire for connections inside the NSG project. However, they do not give information regarding the change in connections among project leaders, participants and neighbourhood residents beyond the project’s scope.   Question 16 “did your project occur in multiple sessions” gives information about the quantity but not about the quality or type of engagement. Projects that occur more than once might create deeper connections when they gather the same people multiple times (e.g. community garden), or might create initial connections when they involve different people each time (e.g. pop-up arts performances). However, this type of changes cannot be measured through the current question. The actions in Table 3 provide a guide for new questions that could be designed to measure this goal. Table 3: NSG participants are connected and engaged when they: I. Connect and Engage Neighbourhood Residents Abundant Community Model  VF Staff and NSG coordinators Project Leaders Surveys  Are members of, or join, local groups  Have close relationships with their neighbours  Take care of their neighbourhood’s residents, especially the vulnerable in their community  Connect their NSG projects to existing community groups, or other NSG projects  Involve neighbours who were not previously involved  Replicate or expand their projects the following year  Continue projects beyond the scope/timeframe of the NSG program  Say “Hi” to their neighbours  Know their neighbours  Develop relationships with RAC members, project leaders or project participants  Do more activities with their neighbours  Make new friends  Join existing programming from partner organizations  Increase their awareness about neighbourhood events and programs  Increase their participation in community activities (e.g. Volunteering, participating in neighbourhood planning processes, joining PAC or other committees)  Make new friends or contacts that remain after the project is finished  Develop a regular group or event as a result of their NSG project  Increase their participation at a local level (joining city committees and planning processes, participating in local elections)   34   3. Clarify the role of “creating deeper connections” in this goal: A shared understanding about the actions associated with each component of the goal will offer a robust framework for grant decisions-making and program promotion. An aspect that needs special attention when addressing this goal is to define whether deepening relationships is associated with connecting or engaging with people in a neighbourhood.  1. Increase emphasis on the idea that “everyone has valuable skills and knowledge to share”: Long-standing NSG coordinators brought to attention the importance of reminding NSG participants that everyone has something to share. Emphasizing this aspect could attract new people to participate in the program.   2. Clarify if the development of skills is an element of this goal: Developing people’s skills is not central for this goal according to “Abundant Community” and Vancouver Foundation’s staff. However, some NSG coordinators mentioned learning new skills as the core aspect of this goal. It is worth discussing this issue further with NSG coordinators in order to arrive at a common understanding about the goal.  II. Share Residents’ Skills and Knowledge Table 4: NSG participants share their skills and knowledge when they: Abundant Community Model  VF Staff and NSG coordinators Project Leaders Surveys  Provide for themselves much of what they require instead of purchasing it  Combine their individual skills in creative ways  Invite other residents to, voluntarily, contribute their skills and knowledge  Offer their skills and knowledge to support their neighbours   Make an inventory of project participants’ skills and interests, and how they could contribute to the project  Contribute their skills and knowledge to organizing an NSG event  Share their skills with others through workshops and mentorships  Look for free resources available in the neighbourhood before they purchase them  Learn skills from their neighbours  Encourage others to share their skills  Support each other with their recently discovered skills and knowledge (e.g. local resident offers a concert for a block party, or does a carpentry job for their neighbour)  Invite community members to get involved in NSG projects  Invite community members to teach their skills  Create a space where residents with a particular expertise can teach other residents   35  3. Define if exchanges mediated by monetary transactions are considered sharing: the revision of this goal could include a consideration of the role of honorariums in the NSG program.  Suggested questions for this goal:  What did you learn about your neighbours’ skills and knowledge? Do you have any ideas about ways in which their skills could be integrated in your project? Or connected among them? What are the ways in which your project and yourself can access/use these skills in the future? (McKnight & Block, 2010; p. 143)  1. Re-examine the role that ownership and pride have in the NSG program: This research made evident that this goal is loosely defined; it even suggested that ownership and pride could be prerequisites for or results of people’s involvement in the program. This issue requires a deeper investigation from Vancouver Foundation.   2. Define the exact meaning and scope of the term “local resource”: the component “use of local resources” includes people and culture as resources. For this reason the responses to question 22, regarding use of local resources, overlapped with responses to questions from goals II (Skills and knowledge) and IV (diversity). The relationship of the “local resources” component with the overall “Build sense of ownership and pride” goal is not clear either. It is not evident whether the use of local resources is an activity that builds ownership and pride, is a display of these sensations, or is a consequence of having pride and ownership in the neighborhood. III. Build Sense of Ownership and Pride within Communities Table 5: NSG participants have ownership and pride when they: Abundant Community Model VF Staff and NSG coordinators Project Leaders Surveys  Do things voluntarily for their community  Feel satisfied with where they live  Feel responsible towards their neighbourhood  Get together to improve the look of their neighbourhood  Create a space where other residents can give back to the community  Make use of local institutions and businesses  Believe they can make a difference in their neighbourhoods  Engage in place-making projects  Use local spaces for their NSG projects  Make use of the talents and skills of local people  Lend their personal belongings for an NSG event  Get in kind donations and support from local businesses  Share their culture with each other   36  Furthermore, it is important to clarify the meaning of “local” in the NSG context because the term is used ambiguously. When referring to “local” resources people mentioned activities that took place within the provincial, municipal and community boundaries. In the NSG context, local could be defined as a resource that is loaned, donated, contributed or sold by a neighborhood resident (it might even include stores owned by neighborhood residents).  3. Identify the relationship between place-making, beautification and neighborhood pride: Beautification and place-making projects are the type of projects that are mostly associated with this goal. Again, it is necessary to make explicit the connection of these types of projects with neighborhoods pride, and if there are particular reasons why physical projects are highlighted over other projects that also generate pride such as block parties.  4. Define the expression “meaningful contribution”: Another action that would clarify this goal further would be to establish the characteristics of an NSG project that makes a “meaningful contribution”. It is important to define these projects, as opposed to those that do not make a meaningful contribution and differentiate the term “making a meaningful contribution” from the similar concept “share skills and knowledge”. The current definition of this component does not offer a clear way of differentiating between NSG projects in terms of how much they achieve this goal.  1. Make more explicit the “accessible” and “barrier-free” components: NSG coordinators stated the importance of taking into account social inequalities in this goal. This means explicitly striving for inclusive and barrier-free projects as part of this goal. Due to the diversity of NSG participants, some of them might not be familiar with the types of concepts and practices related to social inclusion. Vancouver Foundation and partner organizations could explore the types of resources that are needed to strengthen the inclusivity of NSG projects. Another action to promote inclusivity could be to use the NSG evaluation to inform participants about the actions involved in creating accessible/barrier free-events.  2. Define what “highlight diversity” means: Defining this term would make question 23 “Did your project highlight the diversity of your neighbourhood” more clear for survey respondents.   3. Explore if certain areas of diversity need to be respected and celebrated further: Responses from project leaders to this goal often pointed to cultural and age diversity. A multiple choice question that asks if project leaders took measures to facilitate the inclusion of different types of diversity (e.g. sexual orientation, physical ability), could be created. This IV. Respect and Celebrate Diversity  37  questions would focus on the actions of project leaders to respect specific types of diversity instead of enquiring about the diversity of project participants -the current question about the extent to which the project connected people from different backgrounds could be maintained. On one hand, this would increase            awareness about all the ways in which people are different. On the other hand, it would indicate to Vancouver Foundation and NSG coordinator if it is necessary to promote more directly the inclusion of particular social groups in the NSG program.             Table 6: NSG participants celebrate and respect diversity when they: Abundant Community Model VF Staff and NSG coordinators Project Leaders Surveys  Feel they can be themselves without being criticized by their community  Welcome every person, even if they do not know them  Invite everyone to participate, especially those who are labeled by society as deficient    Get together with people from diverse walks of life (age, culture, sexual orientation religion, etc.)  Gather with people they do not usually relate to  Congregate with people that share their live experiences with the purpose of supporting each other   Welcome everyone to their projects, even those who were not invited (e.g. people walking by)  Acknowledge the territory (i.e. Coast Salish) where they live.  Translate invitations and events into languages commonly spoken other than English  Cook dishes from their own culture and share them with others  Provide appropriate food for people with dietary restrictions  Avoid requesting money for attending community events  Give central roles to diverse members of the community   38  This report identified a series of actions that have the potential of increasing the efficiency of the NSG evaluation. A key action would be to reduce the length of current questionnaires; some of the strategies that could support this action are: removing the redundant questions, removing the questions that do not provide new information, reducing the number of open-ended questions, increasing the number of closed-ended questions and alternating the evaluation focus for each year. The implementation of these strategies could simplify and streamline the NSG evaluation process.  A secondary outcome of this report was the identification of the evaluation priorities of Vancouver Foundation and community partners. The NSG evaluation has served as a tool for gathering feedback to adjust the program and to identify the program’s outcomes. This evaluation has also been used for collecting successful and inspiring project stories to support program promotion. Overall, program’s stakeholders are satisfied with the information they obtain from the NSG evaluation.   The guiding principles that orient the design of the evaluation surveys are well established. Program goals and their attainment have a central place in the NSG program evaluation. A strength of the NSG goals lies in the fact that they are grounded in the “Assed-based Community Development” theory as outlined in the book Abundant Community by McKnight and Block (2010). Evidence of goal’s achievement was found in the evaluation data for the 2014 granting cycle; project leaders told multiple stories about ways in which their projects achieved the program’s goals.  Finally, this report presented various sets of definitions of the NSG goals, which can be used by the program’s stakeholders to further clarify the meaning of each NSG goal. The program’s stakeholders have a shared understanding and definition of the main goal of the NSG program: connect and engage neighbourhood residents. In addition, they recognize and have a general understanding of the other program goals: share residents’ skills and knowledge, build a sense of ownership and pride, and respect and celebrate diversity. However, the definition and conceptual boundaries of the goal “build sense of ownership and pride” needs a closer examination from Vancouver Foundation and other program’s stakeholders. Some aspects of each goal require further definition in order to reach a common understanding of the ends pursued within each goal.  Program evaluation is a key component on the growth and development of any program. The Neighbourhood Small Grants program is committed to using the evaluation data to adjust the program in ways that better suit the needs of all its stakeholders. This report aims to offer valuable insights that support Vancouver Foundation’s efforts to constantly improve its evaluation processes.  Concluding Remarks  39  References Adato, M. (2012). Combining quantitative and qualitative methods for program monitoring and evaluation: Why are mixed-method designs best? Washington, DC: World Bank. Berk, R., & Rossi, P. (1999). Thinking about program evaluation (2nd; 2. ed.). Thousand Oaks: Sage Publications.  Couper, M. P., Traugott, M. W., & Lamias, M. J. (2001). Web survey design and administration. The Public Opinion Quarterly, 65(2), 230-253. De Vaus, D. (1991). Surveys in social research (3rd ed.). North Sydney, Australia; London: UCL Press. Denzin, N. K., & Lincoln, Y. S. (2000). The handbook of qualitative research (2nd ed.). Thousand Oaks: Sage Publications. Estrella, M., Blauert, J., Campilan, D., Gaventa, J., Gonsalves, J., Gujit, I. & Ricafort, R. (2000). Learning from change: Issues and experiences in participatory monitoring and evaluation. London; Ottawa: Intermediate Technology Publications Estrella, M. & Gaventa, J. (1998). Who Counts Reality? Participatory Monitoring and Evaluation: A Literature Review. IDS Working Paper 70 Flick, U. (2002). An introduction to qualitative research (2nd ed.). London: Sage. Fournier, D. M. (1995). Establishing evaluative conclusions: A distinction between general and working logic. New Directions for Program Evaluation, (68), 15-32.  Fowler, F. (1995). Improving survey questions: Design and evaluation. Thousand Oaks: Sage Publications. Greene, J. (2000). Challenges in practicing deliberative democratic evaluation. New Directions for Evaluation, 2000(85). Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, Calif: Sage Publications. Horton, D., Alexaki, A., Bennett-Lartey, S., Brice, K., Campilan, D., Carden, F. & Watts, J. (2003). Evaluating capacity development: Experiences from research and development organizations around the world. The Hague: Technical Centre for Agricultural and Rural Cooperation; International Development Research Centre; International Service for National Agricultural Research. Lewis, D. (1994). A Guide to Program Evaluation for Accountability in Non-profit Organizations. Kitchener, ON: LAMP Consultants McDavid, J. & Hawthorn, L. (2006). Program Evaluation & Performance Measurement: An introduction to Practice. Thousand Oaks: SAGE Publications McKnight, J. & Block, P. (2010). The abundant community: Awakening the power of families and neighborhoods. San Francisco, CA; Chicago, Ill.: American Planning Association. Mertens, D., & Wilson, A. (2012). Program evaluation theory and practice: A comprehensive guide (1st ed.). New York, NY: Guilford Press. Rao, V. & Woolcock, M. (2003). Integrating Qualitative and Quantitative Approaches in Program Evaluation. In F. Bourguignon and L. A. Pereira da Silva (Eds.), The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Tools and Techniques, (pp. 165–90). Washington, DC: World Bank. Rogers, P. & Owen, J. (2001). Program evaluation. London: SAGE Publications Ltd. Scriven, M. (1993). Evaluation and Critical Reasoning: Logic’s Last Frontier? In R. Talaska (ed.), Critical Reasoning in Contemporary Culture. Albany: State University of New York Trochim, W. M. K. (1998). An evaluation of michael scriven's “Minimalist theory: The least theory that practice requires”. American Journal of Evaluation, 19(2), 243-249. Weisberg, H., Krosnick, J., & Bowen, B. (1989). An introduction to survey research and data analysis (2nd ed.). Glenview, Ill: Scott, Foresman.    40  APPENDIX A Interview Questionnaire for Vancouver Foundation  About the NSG Program and its goals  1. Currently, what is the main goal of the NSG program  2. Has this goal changed or transformed over the time?  3. Could you explain to me, in your own words, each one of the program’s objectives? - Connect (define) and engage (define) neighbourhood residents - Share residents’ skills and knowledge - Build sense of ownership (define) and pride (define) within a community - Respect and celebrate diversity  4. From your perspective, which is the core component of the program that you consider will remain in the future?  About NSG Program Evaluation   1. Why did you choose survey/questionnaires as the evaluation method  2. Did you consider selecting a different method, interviews or focus groups? Please explain  3. How do you use the information from the evaluation questionnaires?    41  APPENDIX B Interview Questionnaire for NSG Coordinators  About the NSG Program Goals  1. Could you explain to me in your own words each one of the program’s goals? - Connect and engage neighbourhood residents - Share residents’ skills and knowledge - Build sense of ownership and pride within a community - Respect and celebrate diversity  2. From your perspective, which are the two most important goals of the NSG program for your neighbourhood? Why?  About NSG Program Evaluation 3. In general, what is your opinion about the NSG evaluation questionnaires and approach?  4. Are the evaluation questionnaires useful for your organization? How do you use them?  5. Which sections or questions in the questionnaires do you find more useful? a) Project leaders questionnaire b) RAC members questionnaire  6. Which sections or questions in the questionnaires do you find less useful? a) Project leaders questionnaire b) RAC members questionnaire  7. Is there any topic or question that you would like to add to the questionnaire?  8. Do you do your own monitoring or evaluation of the program for your organization? Could you share with me the questions you ask or tell me what topics you focus on in the evaluation?    42  APPENDIX C  2014 NSG Evaluation Questionnaire for Project Leaders  Part I: NSG Project Description   The questions in this section will help us understand the nature of your project. We understand that projects may not turn out as originally intended, so please respond to the questions below based on what actually happened in your project.  1.  Please indicate which NSG neighbourhood you live in.  Burnaby  Central Vancouver  Downtown Peninsula  East Vancouver-Cedar Cottage, Hastings Sunrise and Renfrew  East Vancouver-Carnegie, Raycam and Strathcona  New Westminster   North Shore  North Surrey  Richmond  Westside Vancouver  South Vancouver and Collingwood  2. Project Title or Name:    3. How many people helped you organize the project?    4.  How many people attended your NSG project?   5.  Where did your project take place?   Home  Drive way  Street  Public Park  Community Centre  Neighbourhood House  Other  Project didn’t occur o Comment:  6.  Please provide a brief description of your NSG project. (100 words or less)  7.  What worked well with your NSG project? (100 words or less)  8.  What were the challenges of your NSG project? (100 words or less)  9. Were you a participant in other NSG Projects before you became a Project Leader?    43  10. What inspired you to lead a project?   Part II: Project Support  Many NSG Projects benefit from resources contributed by Project Leaders, participants, and others in the community. These may include resources such as food, materials, tools, space, services, etc.  11. Which of the following types of Local Neighbourhood Support were loaned or donated to your project? (Please check all that apply)   Materials   Tools   Food/Refreshments   Services  Space  Cash donations(Amount)  Others (Please describe)  Part III: NSG Program Support  In 2014, Vancouver Foundation (VF) along with NSG Partner Organizations from participating communities provided resources to assist community members when applying for grants and also to grant recipients or Project Leaders in implementing their funded projects. Your responses to the questions below will help us determine how these resources supported your project and whether there are improvements we can make in the future.   12.  Please rate the following NSG program supports you received as a Project Leader? (Please check all that apply)  Types of supports N/A or didn’t attend Not useful Useful Information session and grant writing support    Grantees’ orientation    Community engagement workshop facilitated by Dr. Collin van Uchelen    Heart of Belonging workshop facilitated by Dr. Collin van Uchelen    NSG Summit (International Settlement Conference and Finding Home workshop)    Support and mentorship provided by the NSG Coordinator and/or Resident Advisory Committee (RAC_ members    NSG Website    Online application    Posting NSG events and project stories on the NSG website    Ability to post events and project stories on NSG Website    Other (Please specify)     13.  Please describe how the above supported your project? (200 words)  14.  What other support would be helpful to help you to plan and implement your project successfully?  Part IV: NSG Program Expectations  One of the responsibilities of an NSG Project Leader is to conduct their project in accord with NSG Guidelines. These include involving others in your project, using funds appropriately, and documenting your project etc..   15. Which of the following activities did you implement in your role as Project Leader?    44  Project leader’s roles  Yes No N/A Project was implemented with support of a co-applicant and/or other helpers     Project was promoted to neighbourhood residents    Project was free    Project met municipal bylaws and permit requirements, if applicable    Grant funds were used to cover project costs      Unused grant money was  returned to the Program Coordinator    Vancouver Foundation support to the project was acknowledged    Project story, photos, and/or videos were shared with Vancouver Foundation and/or the NSG Partner Organization     Project has made use of existing local materials and resources (e.g., borrowing of tools)    Project was or will be completed before the deadline date of November 30, 2014     Part V: NSG Goals and Objectives  The Goals and Objectives of the Neighbourhood Small Grants Program highlight important areas of interest to the Vancouver Foundation. Your project was funded because of its potential to meet one or more of the program goals. This was indicated on your application form. However, we recognize that NSG Projects do not always unfold as planned. At this point in time, we would like you to reconsider the Goals and Objectives. Please indicate which of the following occurred as a direct result of your project.    a) Connect and Engage Neighbourhood Residents  16.  Were participants in your project introduced to one another? [Yes/no]  17.  Did your project occur in multiple sessions? [Yes/no]  18. Would the people who connected through your project likely see each other again? [Yes/no] Please describe (100 words or less)  19. Has your involvement in the NSG Program increased your participation in community activities, groups, and/or affairs? Please describe. (100 words or less)  17. Share Residents’ Skills and Knowledge  20.  a) Did your project involve an expert teaching participants? [Yes/no] b) What skills or knowledge were taught?   21.  a) Did your project help participants to learn from each other? [Yes/no] b) Please provide an example.  c) Build Sense of Ownership and Pride in Communities  22.  a) Did your project make use of local resources in the community such as people, places, and cultures? [Yes/no] b) Please describe.  23. a) Do you think your project enhanced sense of ownership and pride within the local community? [Yes/no] b) Please describe  d) Respect and Celebrate Diversity   45   Diversity is the involvement of people from different age groups, ethnicities, genders, income levels, sexual orientation, physical abilities etc.  24.  a) Did your project highlight the diversity of your neighbourhood? [Yes/no]     b) Please provide an example  25.  a) Did your project connect people from different backgrounds that wouldn’t have met otherwise (e.g., seniors with youth)? [Yes/No] b) Please provide an example.  26. a) Was your project accessible (barrier free) and inclusive (welcoming) to participants? [Yes/No]   e) Create Lasting Impact  27. a) Has your project helped create a lasting resource in your community (e.g., garden, bench, book exchange box, etc.)? (Yes/No)      b) Will the resource be maintained over time (e.g., tending the garden, refinishing when needed, cleaning as necessary, etc.)? (Yes/No)  c) Please describe  28.  a) Is your project likely to occur again in the future? [Yes/No]  b) Please describe.  29.  a) Do you think your project helped build trust between neighbours?   b) Please describe.  30. a) Have you noticed any new changes in your neighbourhood that were created as a result of your project? b)  What are these changes? (75 words or less)  Part V: Personal Impact   In the previous section, we focused on the impact that your project has had on the community or neighbourhood. Here, we would like to know about your project’s impact on your own life.   31.  Has leading an NSG Project changed your relationship to your community?   Connection to your neighbours Increased Decreased No change  Belonging to your neighbourhood     Trust of others in your neighbourhood          Hopeful about your neighbourhood     Empowered to address issues in your neighbourhood     Other (Please describe)          32. Given all you have done as an NSG Project Leader this year, please describe your most memorable experience.    33. Additional comments:   On behalf of the Vancouver Foundation and the Neighbourhood Small Grants Program, we want to thank you for leading a project that contributes so meaningfully to the lives of others. We hope that you and all of those involved in your project feel more connected with one another and more deeply engaged within the community. We trust that you will become an inspiration for others to lead projects in the future. Your involvement is a gift to your community. Thank-you!    46  APPENDIX D  2014 NSG Evaluation Questionnaire for RAC Members  Part I: About You The following four demographic questions will help us understand if we are reaching the diverse communities in our neighbourhoods.  1. Gender  Male  Female  Other  2. Age Range  11 to 17  18 to 24  25 to 34  35 to 44  45 to 54  55 to 64  65 to 74  75 or older  3. Were you born in Canada?   Yes   No   4. How long have you lived in your neighbourhood/NSG Funding Area?  Part II: Resident Advisory Committee Involvement The following questions are about your involvement in the Resident Advisory Committee. We begin with questions about the steps you took prior to joining your RAC. Then, we ask about the types of activities you participated in as a RAC member. We complete this part of the survey with some questions about how well your committee worked over the past year.   5. Before joining the RAC, were you a NSG project leader or NSG project participant in an NSG project? (Yes/No)  6. Were you    a) NSG Project leader   b) NSG project participant   c) Both   d) None  7. How long have you been a RAC member?    8. Are you satisfied with the number of RAC Members in your committee (Yes/no)? If not, what would you suggest?   47  9. The role of being a RAC member includes several kinds of activities. Which of the following activities did you perform in your role as a RAC member?  RAC roles  Yes No N/A Promoted the NSG program through word of mouth, e-mail, social media, posters, etc.    Reviewed grant proposals and made recommendations based on NSG goals, objectives, and guidelines    Assisted in community gathering events such as cheque-issue presentation and wrap-up celebration    Supported Project Leaders by answering questions, attending events, and providing mentorship     Shared stories about noteworthy projects with other RAC members, Coordinator, and VF    Provided feedback and reflection on the NSG program    Made recommendations for potential RAC members from the community    Other (Please describe)     10. What worked well in your committee?  11. What were the challenges in your committee?  12. How do you feel about the granting decisions your RAC made?    13. Were there occasions when you questioned a decision that your RAC was making? If yes, please describe.   14. Did you feel confident to express your opinion and decisions even if they  were different from opinions of other committee members?   15. Are the NSG program goals, objectives and guidelines helpful in making decisions? (Yes/No) Please describe.   16. Are the Greenest City NSG program goals, objectives and guidelines helpful in making decisions (Yes/No) Please describe.  17. What is your most memorable experience as a RAC member?  Part III. Program Support In 2014, Vancouver Foundation provided a variety of resources to RAC members to fulfill their roles. We would like to know if these resources were helpful. Your assessment will help us determine how these resources supported you in your work and whether there are improvements we can make in the future.   18. Please rate how helpful each of the following was to you as a RAC member.      Types of supports  N/A Not useful Useful  RAC Orientation    Access to the RAC manual    Community Engagement Workshop facilitated by Dr. Collin van Uchelen    Heart of Belonging workshop facilitated by Dr. Collin van Uchelen     48  Support/mentorship by the NSG Coordinator    NSG Summit (International Settlement Conference and Finding Home workshop)     Refreshments and meals  during meetings    NSG promotional materials    Access to grant applications online    Ability to make comments and ask questions about applications online    Other (please describe)     19. What other support would be helpful to you in your role?    20. How well were you supported by the NSG Coordinator?  Excellent  Good  Average  Fair  Poor Comment:  Part IV: Personal Impact and Capacity  This section focuses on how your involvement with the RAC has had an impact on yourself and your relationship to your neighbourhood.   21. As a result of your involvement within the RAC, have you  (Yes/No) Please describe a) Met new people    b) Interacted with other RAC members on more than one occasion    c) Interacted with Project Leaders    d) Became more involved in community activities, groups, and/or affairs?      22. Has your involvement within the RAC enhanced your knowledge (valuable information) for carrying out the NSG Program? (Yes/No) If yes, please give an example.   23. Has your involvement within the RAC enhanced your skills (practical abilities) for carrying out the NSG Program? (Yes/No) If yes, please give an example.   24. Has your involvement within the RAC changed your relationship to your community? (Please rate the following)   Connection to your neighbours Increased Decreased No change  Belonging to your neighbourhood     Trust of others in your neighbourhood     49   Hopeful about your neighbourhood     Empowered to address issues in your neighbourhood     Other (Please describe)      Part V: Looking Forward Membership on a Resident Advisory Committee involves meaningful participation in most aspects of the NSG Program. Your experience to date provides you with a unique perspective on the program as a whole.   25. Considering what you know, what suggestions do you have that would improve the overall program.    50  APPENDIX D  NSG Coordinators’ Comments about Project Leaders Questionnaire  Question Number/topic Coordinators’ Comments Part I: NSG Project Description 1. Neighbourhood  2. Project title “Very often applicants don’t remember the title of the project so it would be good if they could log in into their account and see the application and answer the survey through the NSG website”  3. People helped “I like to know how many people organized the event” 4. People attended “This is very useful because the city asks me how many people attended in total” “We always like to know how many people attended the events” 5. Project place “The project didn’t occur should take you to another set of questions to identify why the project didn’t occur. This is kind of a weird place to have this question ” “These are things that I already know” 6. Project description “We have that already [from the application]. Unless they changed it, I don’t think we need it” “This would only necessary only if they changed their project” “These are things that I already know” 7. What worked well “One of the he most important things I need to know” “Interesting” “I like this one” “This is important’ “This ones are quite useful because once you know what work well and what didn’t, you can try to figure out the solutions of those problems.  8. Challenges “One of the he most important things I need to know” “Interesting” “I like it” 9. Former participation in NSG  10. Inspiration “Yes, [it is useful] so I can put it on the promotion as well” “This is useful. Because maybe I can recreate that inspiration to inspire someone else so that they will join [the NSG program]” “I also like the “what inspired you to lead a project?” one. Because it is helpful to know that so you can inspire other people to do it. Specially, since you are trying to get it of the ground in other areas” “I think that this is useful” Part II: Project Support 11. Types of local support “It really doesn’t make a difference to me if a person got stuff donated or not” “It would be interesting to identify which organization gave them donation. Because then we could build a data base of places to go ” “I would say this is less important for me” Part III: NSG Program Support 12. NSG supports - rate “This is all I work for, so I would love to know what is useful or not to them” “The resources offered are different in every neighbourhood so these questions don’t apply to every neighbourhood or every person (e.g. NSG Summit)” “Number 12 is important, what they found useful and didn’t.” 13. NSG supports - how  14. NSG support – other suggested “I think I would like to know what kind of supports people need. It would like to have a concrete A, B, C, D question because it would  51  allow me to see what is the area I need to work on. . For example, some response options that could be included could be: help with promotion, how to get discounts for NSG projects, having/finding volunteers for project, how to contact media, how to get a city permit, connecting project leaders among themselves.” “This is very useful” PART IV: NSG Program Expectations 15. Activities implemented (expectations) “They are not going to say no if they didn’t.”  “I don’t think we have anything that talks about that [use of local materials], not in the application, not in the information. So I don’t know where that question comes from” “to the question “Vancouver Foundation support to the project was acknowledged” add “Vancouver Foundation and partner organization support was acknowledged” “We already know this from attending the events” PART V: NSG Goals and Objectives “I think this questions need to be more specific. We have all kinds of people answering this questions, even literacy issues here.” “These are the kinds of things I like to read about. These are also the kind of things that come up when people share their stories” “I think the answers to these questions are very subjective. And many things that might happen people might not materialize it in their answers” “Just go more brief into the NSG goals and objectives” “maybe create questions with scales to know how much the project met the objectives, instead of yes or no questions” “These questions are good because they are concrete evaluations of the goals of the project” “I think maybe not for me particularly as a program coordinator but section five is important for the project leaders because it gives that reflection piece for them to go “oh, did I do this” 16. Goal I - participants introduced  17. Goal I - multiple sessions  18. Goal I - participants see each other again “This is the kind of question that doesn’t apply to every single project. It applies to small projects but not to big ones” 19. Increased participation  20. Goal II - expert teaching  21. Goal II - learn from each other  22. Goal III – Local resources  23. Goal III - Enhance ownership and pride  24. Goal IV – highlight diversity “What does it mean highlight. Does it mean that all diversity was represented? Or should it be celebrate? Because I also find weird like “we are going to point out all the ways you are different” 25. Goal IV - connect across backgrounds “This is what is important” 26. Goal IV - project accessible “I think that people won’t necessarily know what barrier free is. SO maybe give an example, free, physically accessible” Part VI: Personal Impact “For me this is very important because we want to know how they felt about their NSG projects” 30. Enhanced relations in community “May be replace “hopeful about your neighbourhood” with “excited about your neighbourhood” 31. Memorable experience   


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items