Open Collections

UBC Faculty Research and Publications

Evaluating the impact of a network of research partnerships: a longitudinal multiple case study protocol Hoekstra, Femke; Martin Ginis, Kathleen A; Allan, Veronica; Kothari, Anita; Gainforth, Heather L Nov 12, 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12961_2018_Article_377.pdf [ 854.46kB ]
JSON: 52383-1.0373702.json
JSON-LD: 52383-1.0373702-ld.json
RDF/XML (Pretty): 52383-1.0373702-rdf.xml
RDF/JSON: 52383-1.0373702-rdf.json
Turtle: 52383-1.0373702-turtle.txt
N-Triples: 52383-1.0373702-rdf-ntriples.txt
Original Record: 52383-1.0373702-source.json
Full Text

Full Text

STUDY PROTOCOL Open AccessEvaluating the impact of a network ofresearch partnerships: a longitudinalmultiple case study protocolFemke Hoekstra1,4* , Kathleen A. Martin Ginis1,4,5, Veronica Allan2, Anita Kothari3 and Heather L. Gainforth1,4AbstractBackground: Conducting and/or disseminating research together with community stakeholders (e.g. policy-makers,practitioners, community organisations, patients) is a promising approach to generating relevant and impactfulresearch. However, creating strong and successful partnerships between researchers and stakeholders is complex.Thus far, an in-depth understanding of how, when and why these research partnerships are successful is lacking.The aim of this study is to evaluate and explain the outcomes and impacts of a national network of researchers andcommunity stakeholders over time in order to gain a better understanding of how, when and why researchpartnerships are successful (or not).Methods: This longitudinal multiple case study will use data from the Canadian Disability Participation Project, alarge national network of researchers and community stakeholders working together to enhance communityparticipation among people with physical disabilities. To maximise the impact of research conducted within theCanadian Disability Participation Project network, researchers are supported in developing and implementingknowledge translation plans. The components of the RE-AIM framework (reach, effectiveness, adoption,implementation and maintenance) will guide this study. Data will be collected from different perspectives(researchers, stakeholders) using different methods (logs, surveys, timeline interviews) at different time points duringthe years 2018–2021. A combination of data analysis methods, including network analysis and cluster analysis, willbe used to study the RE-AIM components. Qualitative data will be used to supplement the findings and furtherunderstand the variation in the RE-AIM components over time and across groups.Discussion: The outcomes, impacts and processes of conducting and disseminating research together withcommunity stakeholders will be extensively studied. The longitudinal design of this study will provide a uniqueopportunity to examine research partnerships over time and understand the underlying processes using a variety ofinnovative research methods (e.g. network analyses, timeline interviews). This study will contribute to opening the‘black box’ of doing successful and impactful health research in partnership with community stakeholders.Trial registration: Open Science Framework: Research partnership, Integrated knowledge translation, Stakeholder engagement, Research co-production, Participatory research, Mixed-methods, Impact assessment, People with physical disabilities,practitioners, Health system leaders* Correspondence: femke.hoekstra@ubc.ca1School of Health and Exercise Sciences, University of British ColumbiaOkanagan, Kelowna, BC, Canada4International Collaboration on Repair Discoveries (ICORD), University ofBritish Columbia, Vancouver, BC, CanadaFull list of author information is available at the end of the article© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (, which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver( applies to the data made available in this article, unless otherwise stated.Hoekstra et al. Health Research Policy and Systems          (2018) 16:107 process of transferring research findings to practiceand policy is complex and often unsuccessful [1–4]. Ifresearch findings are not translated, community stake-holders (e.g. policy-makers, practitioners, communityorganisations, patients) cannot benefit from the bestavailable knowledge and healthcare. As a result, researchcannot have an impact in society. Moreover, lack ofadequate translation results in a large amount of re-search waste in terms of the invested time and money toconduct the research [5, 6]. One of the reasons for this‘knowledge-to-practice gap’ is that the needs and re-search priorities of researchers do not always correspondwith the needs and priorities of those who may benefitfrom the research [7, 8]. Engaging stakeholders in theresearch process has been proposed as a promising ap-proach to close this ‘knowledge-to-practice gap’ [9–12].An example of such an approach is integrated know-ledge translation (KT), in which researchers and stake-holders work collaboratively in the research process [9].The extent to which stakeholders are involved in allphases of the research process can vary from project toproject. Independent of the extent of involvement, itrequires a strong and successful partnership between theacademic researcher(s) and the community (i.e. ‘researchpartnership’) [13].Over the last decades, research partnerships havebecome increasingly popular, as illustrated by the largenumber of literature reviews published on different typesof research partnership approaches (cf. [11, 14–19]).Across the fields of health and social sciences, manyresearch partnerships have been created to conduct re-search together on a broader network level (e.g. [20–22])as well as on an individual project level (e.g. [23–25]).Although conducting research within a research partner-ship is popular, setting up a partnership does not happenspontaneously. Partnership formation is often a complexand lengthy process. In addition, setting up a sustainablepartnership is even more challenging [26]. To date,many studies, including literature reviews, have focusedon identifying hampering and facilitating factors to cre-ating successful research partnerships (cf. [11, 15, 16]).Commonly mentioned barriers include excessive timeinvestment, excessive funding pressures, unclear rolesand/or functions, and poor communication betweenmembers of the partnership [15]. On the other hand,commonly mentioned facilitators include trust, respectand a good relationship among partnership members,shared vision and/or goals, and effective communication[15]. However, an overview of important hampering andfacilitating factors is not enough to understand how,when, with whom and why partnerships are successful(or not) in conducting and/or disseminating research.Understanding these underlying processes is importantto provide effective support and guidance to researchersand stakeholders on working collaboratively, which maysubsequently contribute to more relevant and impactfulresearch [10, 27].Nonetheless, studying the underlying processes of asuccessful partnership is challenging due to its multi-fac-torial character and its context-dependent successes.Despite its complexity, several studies have been con-ducted to start opening this ‘black box’ of successfulresearch partnerships (cf. [14–16, 28–30]). To date, mostof the studies were cross-sectional, using only qualitativemethods [15]. No studies have been found that measureresearch partnership quality and synergy over time.Moreover, the majority of the studies included only onecase (i.e. one partnership), instead of multiple cases.Partnerships and projects are heterogeneous, in itselfillustrating the necessity to study multiple cases. Giventheir common goals and processes, networks in whichresearchers and community stakeholders are workingtogether on different projects and in different partner-ships are an ideal setting to study the underlyingprocesses of partnerships and to understand theirheterogeneity [20–22].Canadian Disability Participation Project (CDPP)The CDPP network is meant to enhance communityparticipation among Canadians with physical disabilities[31]. The research projects within the CDPP networkfocus both on improving and understanding the quantity(i.e. the number of people who participate) as well as thequality of participation (i.e. the quality of peoples’participation experiences and satisfaction [32]). The net-work aims to enhance quantity and quality participationin three areas, namely employment, mobility, and sportand exercise. To date, the CDPP network consists of 31principal researchers and 18 community stakeholders(organisations).Within this network, researchers and stakeholderswork in partnerships to conduct and/or disseminatetheir research with community stakeholders following anintegrated KT approach. A well-trained KT specialistsupports researchers in developing and implementingplans to translate their research findings to practice andpolicy, aiming to maximise the impact of CDPP re-search. The goal of this service is to build KT capacityamong researchers. Although the CDPP partnerships areall KT focused, the degree to which stakeholders areengaged in the research process varies from project toproject. Moreover, while the research projects all focuson participation, they differ in their research scope, topicand design. The heterogeneity in partnership characteris-tics provides an opportunity to gain a better understandingof how, when, with whom and why research partnershipsare successful in conducting and/or disseminating researchHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 2 of 11together, and how this is related to impact. These insightsmay contribute to the development of more effective andsustainable research partnerships, and may help to furtheroptimise the support offered to researchers and stake-holders regarding how to conduct and/or disseminateresearch together.Guiding frameworkThis study will use the RE-AIM framework as a guide tounderstand the outcomes and impacts of the CDPP atthe network and project levels [33]. This frameworkincludes the following five components: Reach, Effective-ness, Adoption, Implementation and Maintenance. TheRE-AIM framework was originally developed to studythe public health impact of interventions and pro-grammes. In this study, we elaborated on the work ofSweet et al. [34], who illustrated how the RE-AIMframework can be used as a guide to evaluate the impactof a large partnership between researchers and commu-nity stakeholders. We will operationalise and measurethe RE-AIM components at different levels (networkand project) and at different moments in time. More-over, we will explain the variation in the RE-AIMcomponents (over time and among partnerships) to gaina better understanding of how to conduct successful andimpactful research within a research partnership.Study aimsThe aim of this study is to evaluate and explain the out-comes and impacts of the CDPP at the broader networklevel, as well as at the level of individual projects, usingthe RE-AIM framework. In doing so, this project willoffer an enhanced understanding of how, when, withwhom and why research partnerships are successful (ornot). More specifically, this study aims to identify howthe RE-AIM components change and vary over time atthe network and project levels. Ultimately, this study willcontribute to further opening the ‘black box’ of doingsuccessful and impactful research in partnership withcommunity stakeholders.MethodsStudy overviewAn overview of the study is depicted in Fig. 1. Atwo-level longitudinal multiple case study design usingquantitative (logs, surveys) and qualitative researchmethods (exit interviews, timeline interviews) will beused. Data will be collected from different perspectives(researchers and stakeholders), and at different timepoints between the years 2018 and 2021. The two-leveldesign consists of (1) the CDPP network (‘one case’) and(2) the projects of research partnerships within theCDPP network (‘multiple cases’).Study populationAt the network level, the study population consists of allresearchers, research trainees and community stake-holders (e.g. decision-makers, patients, communityorganisations) that are affiliated with the CDPP networkduring the study period (2018–2021). To be included,participants (researchers, research trainees and commu-nity stakeholders) need to be 18 years and older, andgive consent to participate in this study.At the project level, the study population consists ofresearchers, research trainees and community stake-holders involved in one or more CDPP research projectsduring the study period (2018–2021). CDPP researchprojects are all projects initiated by one of the principalresearchers of the CDPP network and (partly) funded byCDPP. Inclusion criteria for researchers, researchtrainees and community stakeholders are (1) beingactively involved in one or more CDPP researchprojects, (2) 18 years and older, and (3) giving consentto participate in this study. For the in-depth sub-study(Fig. 1), a selection of partnerships working on a CDPPresearch project will be invited. Only CDPP projects thatare in the dissemination stage (i.e. have results to bedisseminated) will be included. We will use a maximumvariation sampling [35] method to recruit partnershipsthat vary the most with regards to the following charac-teristics: (1) partnership quality and synergy, (2) partner-ship size, (3) geographical spread, and (4) research topicand area. Since researchers, research trainees andcommunity stakeholders can be part of different partner-ships, we will allow them to participate a maximum oftwo times in the in-depth sub-study if they belong tomore than one partnership. We will recruit partnershipsuntil data saturation is achieved.RE-AIM componentsTable 1 presents a complete description and operationa-lisation of the five RE-AIM components used to guidethe study at the network and project levels.ReachAt the network level, ‘reach’ refers to the amount (e.g.number of presentations) and type of information (e.g.scientific versus public) that is sent out from the CDPPnetwork to the community. Changes in the amount andtype of information sent out by the network will bemeasured to assess how the ‘reach’ of the network willchange over time. At the project level, ‘reach’ refers tothe amount and type of information sent out by eachCDPP research project. Throughout the study period,trained research coordinators will be collecting thisinformation (e.g. number of presentations, papers, KTproducts) via research reporting forms and up-to-dateCVs of the involved CDPP researchers. In addition,Hoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 3 of 11administrative project documents, like a KT plan andproject evaluations, which the principal researchers needto provide for each project, will be used. Descriptivestatistics (e.g. means, percentages, frequencies) will beused to describe the change over time and differencesacross sectors, disciplines and projects.EffectivenessAt the network level, the effectiveness componentfocuses on an increase in (1) the multidirectional flow ofknowledge and (2) researchers’ capacity for KT activities.Multidirectional flow of knowledgeThe changes in the multidirectional flow of knowledgeover time (2018–2021), across disciplines (mobility, em-ployment, sport and exercise) and sectors (e.g. academic,community, policy) will be evaluated using networkanalysis [36, 37]. A network analysis is a mathematicaland graphical method of analysing complex, interper-sonal processes and can be used to understand howknowledge flows within networks or organisations. Con-ducting network analyses will provide an opportunity tovisualise the CDPP network structure over time usingsociograms and describe the network’s characteristicsusing several measures at both the network (e.g. density,core-periphery structure) as well as the individual level(e.g. degree, closeness) [38–41]. Logs and projectdocuments will be used to collect information about theresearchers, trainees and stakeholders (contact personsand organisations) involved in each of the CDPP re-search projects. Each year of the study period, a networkanalysis will be conducted and key network levelmeasures will be determined. This analysis will allow usto identify how the CDPP network is changing over timein terms of memberships and collaborations.To understand and explain how the CDPP network isfunctioning and changing over time, a selection of theCDPP researchers and community stakeholders will beinvited for an ‘exit interview’ at the end of the studyperiod (Fig. 1). The exit interviews will focus on re-searchers’ and stakeholders’ experiences with the CDPPnetwork, their view on how the CDPP network haschanged the multidirectional flow of knowledge, andtheir view on how the CDPP network has influenced theKT capacity among researchers, as well as related facili-tators and barriers. A key element of the exit interviewswill be to discuss and reflect on changes in the CDPPnetwork structure over time (2018–2021) [42]. To guidethis discussion, the interviewer will show the inter-viewees the sociograms generated through the networkanalyses. After explaining these sociograms, the inter-viewer will ask the researcher or stakeholder to reflecton changes in the sociograms over time. Moreover, theinterviewer will ask questions about researchers’ orstakeholders’ own position within the CDPP networkand how that position has changed over time. Probingquestions will be asked to gain more insight into how,when and why their position within the CDPP networkstructure has changed or remained stable, in addition toperceived facilitators and barriers.Examples of interview questions and prompts include(1) “Please tell me about your experiences being a partof the CDPP network”, (2) “What are, in your opinion,successes of the CDPP network?”, (3) “What are, in youropinion, challenges of the CDPP network?”, (4) “Howhas the CDPP network structure changed over time, andhow do you explain these changes?”, (5) “How do youfeel about your position within the CDPP network?”,and (6) “What lessons did you learn from being part ofthe CDPP network?”Furthermore, all members of the CDPP network willbe asked to complete an online survey yearly (T0:August 2018, T1: August 2019; T2: August 2020: T3:August 2021) in order to obtain information about theirgeneral demographics, their experiences with theCDPP network, the value of the CDPP network, andthe quality of the communication within the network,as well as questions about researchers’ capacity forKT activities. The survey will be adapted to theparticipant’s role in the network (researcher or stake-holder). Filling out the survey will take approximately10 min.Fig. 1 An overview of the data sources and moments of measurement at the network and project levelHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 4 of 11Table 1 Operationalisation of the RE-AIM components at network and project levelsRE-AIM component General description Operationalisation Data sourceReachNetwork The amount and type of informationthat is sent out to the community fromthe CDPP network yearlyTotal number of academic papers, academicpresentations, community presentations, non-academic papers, policy briefs and reports, articles,blogs and papers on websites, social media posts,mass media press releases, guides, toolkits, videosand aids (KT products) yearlyLogs + projectdocumentsProject The amount and type of informationthat is sent out to the community byeach CDPP research projectMean, ranges and frequencies of academic papers,academic presentations, community presentations,non-academic papers, policy briefs and reports,articles, blogs and papers on websites, social mediaposts, mass media press releases, guides, toolkits,videos and aids (KT products) per projectLogs + projectdocumentsEffectivenessNetwork Multidirectional flow of knowledge: anincrease in the multidirectional flow ofknowledge across disciplines and sectorswithin the CDPP network over timeChanges in the network analysis measures(e.g. density, reciprocity, core-periphery structure)of the CDPP network over timeLogs + projectdocuments, exitinterviewsResearchers’ capacity for KT activities: anincrease in researchers’ capacity for KTactivities before and after the supportfrom the KT consultation service, and anincrease over the time of the studyperiod (2018–2021)Changes in researchers’ capability, opportunity andmotivation to translate their research findings to anon-academic audience before and after theyreceived support from CDPP’s KT consultationserviceChanges in researchers’ capability, opportunity andmotivation to translate their research findings to anon-academic audience over time (2018–2021)Project surveys, annualsurveys, exit interviewsProject Partnership quality and synergy: anoptimisation of partnership quality andsynergy in the process of conductingand/or disseminating research in aresearch partnershipIdentification of partnership profiles based onindicators of partnership quality and synergyChanges in indicators of partnership quality andsynergy over timeProject surveys, annualsurveys, timelineinterviewsAdoptionNetwork The extent to which CDPP researchersdecide to contact the KT consultationservice to translate their researchfindings to an academic andnon-academic audiencePercentage of CDPP researchers that contacted theKT consultation serviceNumber of times that CDPP researchers haveconsulted KT consultation serviceThe total number and duration of the supportprovided by the KT consultation serviceLogs + projectdocumentsProject The extent to which CDPP researchprojects are conducted anddisseminated in a research partnershipPercentage of CDPP research projects that areconducted in partnership with communitystakeholdersPercentage of CDPP research projects that aredisseminated in partnership with communitystakeholdersPercentage of CDPP research projects that areconducted and disseminated in partnership withcommunity stakeholdersPercentage of CDPP research projects that are notconducted or disseminated in partnership withcommunity stakeholdersLogs + projectdocumentsImplementationNetwork The extent to which goals stated by thedirectors and team leads of the CDPPare achieved (i.e. implementation asintended)Conformity of intended network’s goals andachieved goalsLogs, project documents,exit interviewsProject The extent to which the CDPPpartnerships have conducted and/ordisseminated their research according tothe KT plan (i.e. implementation asintended)Conformity of intended KT plan and KT evaluation Logs, project documentsHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 5 of 11Quantitative data collected with annual surveys will bedescribed and tested with appropriate statistics usingIBM SPSS Statistics for Windows Version 24.0. All exitinterviews will be audio recorded and transcribedverbatim. Qualitative data will be analysed thematic-ally to explore trends and patterns in participants’experiences with and perceptions of the CDPPnetwork [43]. The analysis will be conducted usingthe software programme NVivo 11 for Windows.Researchers’ capacity for KT activitiesChanges in researchers’ capacity for KT activities will beassessed before and after receiving support from the KTspecialist and will be assessed over the study period(2018–2021). Researchers’ capacity for KT activities willbe measured using behaviour change theory and will bespecifically focused on researchers’ capability, opportun-ity and motivation (i.e. COM-B [44]) to translate theirresearch findings to a non-academic audience. Accord-ing to the COM-B model, researchers must have per-ceived capability, opportunity and motivation to engagein a certain behaviour [44]. To examine changes inresearchers’ capability, opportunity and motivation fortranslating research findings to a non-academic audi-ence, researchers will be asked to fill out a short surveyat three moments in time, namely pre-survey (t0),post-survey (t1) and follow-up survey (t2). The pre- andpost-surveys will, respectively, be conducted immediatelybefore and after the support from the KT specialist,whereas the follow-up survey (t2) will be conducted 1year later. Furthermore, items related to researchers’capacity for KT activities will be included in the annualsurveys (T0–T3).Qualitative data from the ‘exit interviews’ will be usedto supplement and explain the findings of the surveydata about researchers’ capacity for KT activities usingtriangulation [35]. In addition, we will use the informa-tion about the form, frequency and content of the actualsupport provided by the KT specialist to explain andunderstand if researchers’ KT capacity has been changedover time. The KT specialist will therefore registerdetails about the support by logging the form (internet,phone, face-to-face) and duration of the meetings withthe researchers. In addition, every meeting will be audiorecorded to collect details about the content of themeetings.At the project level, effectiveness refers to an optimisa-tion of partnership quality and synergy in the process ofconducting and/or disseminating research in a researchpartnership (Table 1). Partnership quality is operationa-lised using nine indicators for a successful partnershipidentified by Kothari et al. [45], namely communication,collaborative research, dissemination of research, re-search findings, negotiation, partnership enhancement,information needs, level of rapport, and commitment.Partnership synergy refers to the extent to which thepartnership combines the complementary knowledge,skills, and resources of all members of the partnership tocreate new ideas and look for better ways to solve prob-lems and achieve goals [46–48]. The idea is that, by cre-ating synergy the partnership can achieve more than anyof its individual members and become “a whole that isgreater than the sum of its parts” [48]. Partnership syn-ergy is considered a key indicator for a successful collab-oration [47, 48]. Insight into different partnershipprofiles of indicators and partnership quality and synergywill help to gain a better understanding how, when, withwhom and why research partnerships are successful.To evaluate the effectiveness component at the projectlevel, a selection of research partnerships will be studiedin-depth (Figs. 1 and 2). All key members of the selectedpartnerships (researchers, trainees, stakeholders) will beinvited for an additional in-depth sub-study includingthe follow-up survey (t2) and two timeline interviews(Fig. 2). A timeline interview is a qualitative researchtool that can be used to study individuals’ experiencesover time [49]. More specifically, it is a way to linkpersonal stories or narratives with a broader context. Inthis study, timeline interviews will be used to obtaininformation about researchers’, trainees’ and communitystakeholders’ experiences working together on a prede-termined CDPP research project. We expect the numberTable 1 Operationalisation of the RE-AIM components at network and project levels (Continued)RE-AIM component General description Operationalisation Data sourceMaintenanceNetwork A sustainable multidirectional flow ofknowledge across disciplines and sectorswithin the CDPP networkAnnual survey (T3), exitinterviewsThe long-term capacity among CDPPresearchers for KT activitiesAnnual survey (T3), exitinterviewsProject The continuation of conducting and/ordisseminating research projects in aresearch partnership (i.e. ‘sustainablepartnerships’)Project surveys, timelinesinterviewsCDPP Canadian Disability Participation Project, KT knowledge translationHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 6 of 11of participants per partnership will vary from two to five,depending on the size of each partnership. The in-depthsub-study consists of three parts, namely the follow-upsurvey, an individual timeline interview and a groupinterview (Fig. 2).Part I: follow-up surveyAll members of the selected partnership will be asked tofill out the follow-up survey (t2) to obtain informationabout partnership quality and synergy from a researcherand stakeholder perspective. Partnership quality will beassessed using an adapted version of the Partnership In-dicators Questionnaire [21, 45]. Because the PartnershipIndicators Questionnaire was originally developed in thecontext of policy-maker–researcher partnerships, weadapted it for our sample by piloting the questionnairewith researchers and stakeholders who had experiencewith working together on research projects.Partnership synergy will be assessed using the nineitems of the Partnership Self-Assessment Tool [47, 48].A partnership synergy score will be calculated reflectingthe extent to which the members of a partnership areachieving more together than they can on their own.Completing the survey (pre-survey, post-survey, follow-upsurvey) will take approximately 10 min per survey. Thefollow-up survey (part I) will motivate researchers,trainees and stakeholders to think about their partnershipprior to the individual timeline interview (part II). Inaddition, this survey will help the interviewer to preparethe timeline interviews and identify possible probingquestions.Part II: individual timeline interviewSecond, all members of the selected partnerships will beinvited for an individual timeline interview. During thisinterview, the interviewer and the interviewee willco-create a timeline of the interviewee’s experiences withconducting and/or disseminating research in a researchpartnership, from the first communication aboutworking together on the research project to present. Thetimeline will mark key moments and activities through-out the research process, such as ‘first meeting withpartners’, ‘decision to collaborate’, ‘discussing researchplan’, ‘ethical procedures’, ‘ethical approval’, ‘start of datacollection’, ‘start of data analysis’, and ‘discussion ofresearch findings’. The interviewee will lead the processof adding key moments and activities to the timelineand, if necessary, the interviewer will ask the intervieweefor permission to include other generic milestones thattypically occur throughout the research process (e.g. de-fining the research question). For each moment or activ-ity included on the timeline (i.e. moment in the researchprocess), the interviewer will ask questions such as: (1)“How did you feel during that moment?”, (2) “Describethe relationship between you and your partners?”, (3)“On a scale of 1 to 10, how would you rate your generalsatisfaction level regarding the partnership functioning?”,(4) “What barriers did you perceive at this moment?”,and (5) “What facilitators did you perceive at thismoment?”. Because the interviewer and interviewee willwork together to develop the timeline, the intervieweewill have an active role in the reporting process. As such,the interviewer and interviewee will have shared owner-ship and analytic power during the interview session[49]. All interviews will be conducted by the first author(FH) in person or online using a video conversion inter-face (Vidyo). If interviews are conducted in person, thetimeline will be created on paper. If interviews are con-ducted via internet, the timeline will be created using anonline application. The interviewer may take additionalnotes during the interview sessions. After each session,the created timelines will be shared and verified with theparticipants.Part III: group interviewThird, all key members of the partnership will be invitedfor an interview session together, allowing the inter-viewees the opportunity to interact with one another.Prior to this group interview session, the interviewer willcombine the individual timelines and experiences of allmembers of the partnership into one group timeline.During the group session, all members of the partnershipwill be asked to discuss and reflect on this group timeline.By doing so, the experiences of all partnership membersFig. 2 An overview of the data sources and moments of measurement at the project levelHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 7 of 11will be compared and differences and similarities will bediscussed. The interviewer will ask questions such as (1)“Can you reflect on this moment or activity?”, (2) “Whatwent well?”, (3) “What did not go as expected?”, (4) “Whatwould you do differently if you could do it over?”, (5)“How did your relationship differ between differentmoments on the timeline?”, and (6) “In general, whatlessons did you learn from this partnership?”The last part of this group interview will focus ongeneral indicators for a research partnership. The inter-viewer will ask questions such as (1) “In your opinion,what is a successful research partnership?” and (2) “Howshould researchers and community stakeholders ideallywork together?”. The interview questions will be pilotedand the interview guides will be adjusted accordingly.Data analysisAll interview sessions will be audio recorded andtranscribed verbatim. The transcripts, notes and createdtimelines will be analysed using an inductive thematicanalysis approach to explore trends and patterns inparticipants’ experiences with working in partnership ona CDPP research project [43]. All qualitative data col-lected at the project level (individual timeline interviewsand group interviews) will be analysed collectively. Theanalysis will be conducted following the six steps de-scribed by Braun et al. [43], which include familiarisationwith the data, coding of the data, developing the themes,refining the themes, naming the themes and writing upthe findings. We will illustrate the themes with quota-tions from participants. The analysis will be conductedusing the software programme NVivo 11 for Windows.We will use a flexible approach to determine theappropriate and relevant criteria to evaluate the qualityof our qualitative research methods [50, 51].Following our inductive analysis of the qualitative data,we will conduct a mixed method analysis of the quanti-tative and qualitative data (Parts I–III) to inform a betterunderstanding of how, when, with whom and whyresearch partnerships are successful. The mixed methodapproach will be conducted in two main steps. First, wewill conduct a hierarchical cluster analysis [52] to iden-tify different profiles of partnerships based on indicatorsof partnership quality and synergy measured at thefollow-up survey (t2). Each partnership will be groupedinto a k number of profiles (i.e. clusters). Within each ofthese profiles, partnerships will be most similar to eachother in terms of partnership quality and synergy, butthey will be most different from other profiles (i.e.minimum within-profile variation and maximum be-tween-profile variation). Second, we will determine fac-tors and characteristics of research partnerships that areassociated with the identified partnership profiles. Bothquantitative survey data (annual surveys, pre- andpost-surveys) and qualitative data (timeline interviews)will be used to describe the partnership profiles anddetermine factors and characteristics associated withsuccessful partnerships. For this analysis step, the quali-tative data will be re-analysed using a deductiveapproach focusing on specific factors and partnershipcharacteristics. The identified partnership profiles willhelp us gain a further understanding of how, when, withwhom and why research partnerships are successful (ornot) by explaining the variation between partnershipsand by gaining insights on how partnerships are func-tioning over time.AdoptionAt the network level, adoption describes the extent towhich CDPP researchers decide to contact the KTconsultation service to translate their research findingsto academic and non-academic audiences (Table 1). Atthe project level, adoption describes the extent to whichCDPP research projects are conducted and disseminatedin a research partnership. The logs and project docu-mentation will be used to assess the adoption outcomeat the network and project level.ImplementationAt the network level, implementation focuses on theextent to which the goals stated by the directors andteam leads of the CDPP network are achieved at the endof the study period (i.e. implementation as intended). Atthe project level, implementation focuses on the extentto which the CDPP partnerships have disseminated theirresearch findings according to their KT plan. Therefore,the planned KT activities mentioned in the KT plan willbe compared with the KT activities that are actuallycompleted as reported in the project documents.MaintenanceAt the network level, maintenance refers to a sustainablemultidirectional flow of knowledge across disciplinesand sectors within the CDPP network and the long-termcapacity for KT activities among CDPP researchers. In-formation about the maintenance of the CDPP networkwill be obtained using the annual survey T3 and the exitinterviews (Fig. 2). At the project level, maintenancerefers to the continuation of conducting and/or dissem-inating research projects in a research partnership (i.e.sustainable partnerships). Information about researchers’and stakeholders’ views on the continuation of theirpartnership will be collected during the timeline inter-views (Fig. 2).DiscussionThis paper outlines a protocol to study the outcomes,impacts and processes of conducting and disseminatingHoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 8 of 11research together with community stakeholders at differ-ent levels (network and project), using the RE-AIM frame-work, in the context of research among people withphysical disabilities. The longitudinal design of this studywill provide a unique opportunity to study research part-nerships over time and to understand the underlying pro-cesses using a variety of innovative research methods ( analyses over time, timeline interviews).This study will contribute to the science of KT, imple-mentation, partnerships and impact evaluation. By usingthe RE-AIM framework as a guide to study the impactof a network of research partnerships, we elaborated onthe work of Sweet et al. [34]. Using the RE-AIM frame-work, we will be able to collect, organise and interpretour data in a structural and systematic way. As such, wewill have the capacity to build a rich dataset, therebyincreasing the knowledge base regarding the creation ofeffective and sustainable research partnerships. In thisway, we hope to further open the ‘black box’ of success-fully conducting and disseminating research in partner-ship with stakeholders.In addition to these scientific contributions, this studywill also have practical contributions. The findings ofthis study may be used by many groups working in theareas of health, social sciences and beyond. First,researchers who work or want to work together inpartnership with community stakeholders may benefitfrom the findings of this study. The project level datawill provide new insights into the underlying processesof conducting and/or disseminating research in partner-ship. These insights may guide researchers in makingdecisions regarding how, when and with whom stake-holders should be engaged in their research processes.Moreover, the identified partnership profiles of partner-ship quality and synergy may be used to optimise thetailored support for researchers on how to work in aresearch partnership. More specifically, we will use thefindings of our study to improve and expand the CDPP’sKT support services to further enhance KT capacityamong researchers. Ultimately, our findings may be usedby researchers from all levels as a guide to create andsustain research partnerships, which may substantiallycontribute to closing the ‘knowledge-to-practice gap’within health and social science research.Second, community stakeholders who are or want tobe involved in conducting and/or disseminating researchmay benefit from the findings of this study. Stakeholderswith different backgrounds, experiences, roles and re-sponsibilities will participate in our study allowing us tolearn from this heterogeneity and gain a better under-standing of what works for whom the best. The insightsmay help stakeholders in decision-making processesregarding their engagement in research processes onboth a network and project level. We expect that, ifstakeholders have a better overview of the potential costsand benefits of working together on a research project,more efficient and effective research partnerships can becreated. Furthermore, we hope that our network leveldata will show the value and impact of a largeKT-focused network of researchers and communitystakeholders, such as the CDPP, which may inspireresearchers and stakeholders in other fields and othercountries to set up similar (inter)national networks.Lastly, funding agencies of partnered research maybenefit from the findings of this study. For example, inCanada, partnered research approaches are becomingincreasingly incentivised as evidenced by the grantingopportunities across different federal agencies thatrequire partnership building (e.g. Social Sciences andHumanities Research Council of Canada PartnershipDevelopment and Partnership Grants). Our findings maybe used as a first step in developing criteria for creatingsuccessful research partnerships in health and socialsciences. Such criteria may help funding agencies de-cide how to spend their money. Having such criteriawill also help to monitor and evaluate the fundedpartnered research projects. Furthermore, we arecurrently reviewing and synthesising the literature onresearch partnerships in collaboration with researchersfrom different institutions across Canada [53]. In thefuture, we hope that the findings of this literatureoverview, in combination with the findings of thecurrent study, will be used to develop guidelines forconducting and disseminating research together withcommunity stakeholders. Such guidelines will not onlyhelp funding agencies, but also researchers and stake-holders working together in partnership.In summary, this study will gain a better understand-ing of the underlying processes of research partnershipsworking together on research projects to enhanceparticipation among people with physical disabilities.Therefore, the findings of this study may be relevant fora broad audience, including health researchers, commu-nity stakeholders and funding agencies. Moreover, thisstudy will contribute to opening the ‘black box’ of doingsuccessful and impactful health research in partnershipwith community stakeholders.AbbreviationsCDPP: Canadian Disability Participation Project; KT: knowledge translationAcknowledgementsNot applicable.FundingThis study is supported by a partnership grant from the Social Sciences andHumanities Research Council of Canada (grant no. 895-2013-1021) for theCanadian Disability Participation Project ( HG is funded by aMichael Smith Foundation for Health Research Scholar Award [#16910].KMG holds the Reichwald Family UBC Southern Medical Program Chairin Preventive Medicine.Hoekstra et al. Health Research Policy and Systems          (2018) 16:107 Page 9 of 11Availability of data and materialsThe datasets generated during and/or analysed during the current studywill not be publicly available due to concerns about confidentiality butanonymised group-level datasets will be available from the correspondingauthor on reasonable request after the study findings are published. Relevantresearch materials, such as questionnaires and interview guides, will beavailable through Open Science Framework [54].Authors’ contributionsFH, KMG, VA, AK and HG were involved in the design of the study. KMG andHG contributed to obtaining funding for the study. FH drafted themanuscript. KMG, VA, AK and HG reviewed and provided feedback on themanuscript drafts. All authors read and approved the final manuscript.Ethics approval and consent to participateThis study protocol was approved as a minimal risk study by the ResearchServices Behavioural Research Ethics Board of the University of BritishColumbia Okanagan on July 17, 2018, as H18–00459. Participants in thisstudy will be asked to provide written or verbal consent.Consent for publicationParticipants in this study will be asked to give permission to use thecollected data for scientific publications.Competing interestsThe authors declare that they have no competing interests.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1School of Health and Exercise Sciences, University of British ColumbiaOkanagan, Kelowna, BC, Canada. 2School of Kinesiology and Health Studies,Queen’s University, Kingston, ON, Canada. 3School of Health Studies, WesternUniversity, London, ON, Canada. 4International Collaboration on RepairDiscoveries (ICORD), University of British Columbia, Vancouver, BC, Canada.5Department of Medicine, University of British Columbia Okanagan, Kelowna,BC, Canada.Received: 7 September 2018 Accepted: 7 October 2018References1. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translationof health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–7.2. Green LW, Ottoson JM, Garcia C, Hiatt RA. Diffusion theory and knowledgedissemination, utilization, and integration in public health. Annu Rev PublicHealth. 2009;30:151–74. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translationof research findings. Implement Sci. 2012;7:50. Lomas J. The in-between world of knowledge brokering. BMJ. 2007;334:129–32. Chalmers I, Glasziou P. Avoidable waste in the production and reporting ofresearch evidence. Lancet. 2009;374:86–9. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al.Biomedical research: increasing value, reducing waste. Lancet.2014;383:101–4. Caplan N. The two-communities theory and knowledge utilization. AmBehav Sci. 1979;22:459–70.8. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM,et al. How to increase value and reduce waste when research priorities areset. Lancet. 2014;383:156–65. Canadian Institutes of Health Research. Guide to Knowledge TranslationPlanning at CIHR: Integrated and End-of-Grant Approaches. 2017. Accessed 7 Feb 2018.10. Graham ID, Kothari A, McCutcheon C, Integrated Knowledge TranslationResearch Network Project Leads. Moving knowledge into action for moreeffective practice, programmes and policy: protocol for a researchprogramme on integrated knowledge translation. Implement Sci. 2018;13:22. Jagosh J, Macaulay AC, Pluye P, Salsberg J, Bush PL, Henderson J, et al.Uncovering the benefits of participatory research: implications of a realistreview for health research and practice. Milbank Q. 2012;90:311–46. Kothari A, McCutcheon C, Graham ID. Defining integrated knowledgetranslation and moving forward: a response to recent commentaries. Int JHealth Policy Manag. 2017;6:299–300. Jansson SM, Benoit C, Casey L, Phillips R, Burns D. In for the long haul:knowledge translation between academic and nonprofit organizations.Qual Health Res. 2010;20:131–43. Camden C, Shikako-Thomas K, Nguyen T, Graham E, Thomas A, Sprung J, etal. Engaging stakeholders in rehabilitation research: a scoping review ofstrategies used in partnerships and evaluation of impacts. Disabil Rehabil.2015;37:1390–400. Drahota A, Meza RD, Brikho B, Naaf M, Estabillo JA, Gomez ED, et al.Community-academic partnerships: a systematic review of the state of theliterature and recommendations for future research. Milbank Q.2016;94:163–214. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledgetranslation (IKT) in health care: a scoping review. Implement Sci.2016;11:38. Salsberg J, Parry D, Pluye P, Macridis S, Herbert CP, Macaulay AC. Successfulstrategies to engage research partners for translating evidence into actionin community health: a critical review. J Environ Public Health.2015;2015:191856. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impactthrough co-creation in community-based health services: literature reviewand case study. Milbank Q. 2016;94:392–429. Nwanyanwu KH, Grossetta Nardini HK, Shaughness G, Nunez-Smith M,Newman-Casey PA. Systematic review of community-engaged research inophthalmology. Expert Rev Ophthalmol. 2017;12:233–41. Ginis KA, Latimer-Cheung A, Corkum S, Ginis S, Anathasopoulos P, Arbour-Nicitopoulos K, et al. A case study of a community-universitymultidisciplinary partnership approach to increasing physical activityparticipation among people with spinal cord injury. Transl Behav Med.2012;2:516–22. Kothari A, Sibbald SL, Wathen CN. Evaluation of partnerships in atransnational family violence prevention network using an integratedknowledge translation and exchange model: a mixed methods study.Health Res Policy Syst. 2014;12:25. Nystrom ME, Karltun J, Keller C, Andersson Gare B. Collaborative andpartnership research for improvement of health and social services:researcher's experiences from 20 projects. Health Res Policy Syst.2018;16:46. Van Olphen J, Ottoson J, Green L, Barlow J, Koblick K, Hiatt R. Evaluation ofa partnership approach to translating research on breast cancer and theenvironment. Prog Community Health Partnersh. 2009;3:199. McGrath PJ, Lingley-Pottie P, Emberly DJ, Thurston C, McLean C. Integratedknowledge translation in mental health: family help as an example.J Can Acad Child Adolesc Psychiatry. 2009;18:30–7.25. Eriksson CC, Fredriksson I, Froding K, Geidne S, Pettersson C. Academicpractice-policy partnerships for health promotion research: experiencesfrom three research programs. Scand J Public Health. 2014;42:88–95. Ahmed SM, Maurana C, Nelson D, Meister T, Young SN, Lucey P. Openingthe black box: conceptualizing community engagement from 109community-academic partnership programs. Prog Community HealthPartnersh. 2016;10:51–61. Gagliardi AR, Kothari A, Graham ID. Research agenda for integratedknowledge translation (IKT) in healthcare: what we know and do not yetknow. J Epidemiol Community Health. 2017;71:105–6. et al. Health Research Policy and Systems          (2018) 16:107 Page 10 of 1128. Cargo M, Mercer SL. The value and challenges of participatory research:strengthening its practice. Annu Rev Public Health. 2008;29:325–50. Lavis JN, Robertson D, Woodside JM, McLeod CB, Abelson J, KnowledgeTransfer Study Group. How can research organizations more effectivelytransfer research knowledge to decision makers? Milbank Q.2003;81:221–48 171–2.30. Tricco AC, Zarin W, Rios P, Pham B, Straus SE, Langlois EV. Barriers,facilitators, strategies and outcomes to engaging policymakers, healthcaremanagers and policy analysts in knowledge synthesis: a scoping reviewprotocol. BMJ Open. 2016;6:e013929. CDPP. Canadian Disability Participation Project. Accessed 7Feb 2018.32. Martin Ginis KA, Evans MB, Mortenson WB, Noreau L. Broadening theconceptualization of participation of persons with physical disabilities: aconfigurative review and recommendations. Arch Phys Med Rehabil.2017;98:395–402. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact ofhealth promotion interventions: the RE-AIM framework. Am J Public Health.1999;89:1322–7.34. Sweet SN, Ginis KA, Estabrooks PA, Latimer-Cheung AE. Operationalizing theRE-AIM framework to evaluate the impact of multi-sector partnerships.Implement Sci. 2014;9:74. Green J, Thorogood NP. Qualitative Methods for Health Research. ThousandOaks: Sage; 2013.36. Provan K, Veazie M, Staten L, Teufel-Shone N. The use of network analysis tostrengthen community partnerships. Public Adm Rev. 2005;65:603–13.37. Yousefi-Nooraie R, Dobbins M, Brouwers M, Wakefield P. Informationseeking for making evidence-informed decisions: a social network analysison the staff of a public health department in Canada. BMC Health Serv Res.2012;12:118. Carrington P, Scott J, Wasserman S. Models and Methods in Social NetworkAnalysis. Cambridge: Cambridge University Press; 2005.39. Gainforth HL, Latimer-Cheung AE, Moore S, Athanasopoulos P, Martin GinisKA. Using network analysis to understand knowledge mobilization in acommunity-based organization. Int J Behav Med. 2015;22:292–300. Luke DA, Harris JK. Network analysis in public health: history, methods, andapplications. Annu Rev Public Health. 2007;28:69–93. Valente T. Social Networks and Health: Models, Methods, and Applications.New York: Oxford University Press; 2010.42. Ryan L, D’Angelo A. Changing times: migrants’ social network analysisand the challenges of longitudinal research. Soc Networks.2018;53:148–58.43. Braun V, Clarke V, Weate P. Using thematic analysis in sport and exerciseresearch. In: Smith B, Sparkes AC, editors. Handbook of Qualitative Researchin Sport and Exercise. New York: Routledge; 2016.44. Michie S, van Stralen MM, West R. The behaviour change wheel: a newmethod for characterising and designing behaviour change interventions.Implement Sci. 2011;6:42. Kothari A, MacLean L, Edwards N, Hobbs A. Indicators at the interface:managing policymaker-researcher collaboration. Knowl Manag Res Pract.2011;9:203–14.46. Lasker RD, Weiss ES. Creating partnership synergy: the critical role ofcommunity stakeholders. J Health Hum Serv Adm. 2003;26:119–39.47. Lasker RD, Weiss ES, Miller R. Partnership synergy: a practical framework forstudying and strengthening the collaborative advantage. Milbank Q.2001;79:179–205 III-IV.48. Weiss ES, Anderson RM, Lasker RD. Making the most of collaboration:exploring the relationship between partnership synergy and partnershipfunctioning. Health Educ Behav. 2002;29:683–98. Adriansen H. Timeline interviews: a tool for conducting life history research.Qual Stud. 2012;3:40–55.50. Sparkes AC, Smith B. Qualitative research methods in sport, exercise andhealth: from process to product. New York: Routledge; 2014.51. Smith B, McGannon KR. Developing rigor in qualitative research: Problemsand opportunities within sport and exercise psychology. Int Rev Sport ExercPsychol. 2018;11(1):101–21. Ward JH. Hierarchical Grouping to optimize an objective function. J Am StatAssoc. 1963;58:236–44.53. Hoekstra F, Mrklas KJ, Sibley K, Nguyen T, Vis-Dunbar M, Neilson C, et al.Understanding collaborative approaches to research: A synthesis of theresearch partnership literature. Open Science Framework; 2018. Hoekstra F, Allan V, Gainforth HL. The impact of partnerships betweenresearchers and community stakeholders: how, when and why are theysuccessful? Open Science Framework; 2018. et al. Health Research Policy and Systems          (2018) 16:107 Page 11 of 11


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items