UBC Faculty Research and Publications

Understanding the performance and impact of public knowledge translation funding interventions: Protocol… McLean, Robert K D; Graham, Ian D; Bosompra, Kwadwo; Choudhry, Yumna; Coen, Stephanie E; MacLeod, Martha; Manuel, Christopher; McCarthy, Ryan; Mota, Adrian; Peckham, David; Tetroe, Jacqueline M; Tucker, Joanne Jun 22, 2012

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13012_2012_Article_514.pdf [ 430.4kB ]
JSON: 52383-1.0167791.json
JSON-LD: 52383-1.0167791-ld.json
RDF/XML (Pretty): 52383-1.0167791-rdf.xml
RDF/JSON: 52383-1.0167791-rdf.json
Turtle: 52383-1.0167791-turtle.txt
N-Triples: 52383-1.0167791-rdf-ntriples.txt
Original Record: 52383-1.0167791-source.json
Full Text

Full Text

STUDY PROTOCOL Open AccessUnderstanding the performance and impact ofpublic knowledge translation fundinginterventions: Protocol for an evaluation ofCanadian Institutes of Health Researchknowledge translation funding programsRobert K D McLean1*, Ian D Graham2,3, Kwadwo Bosompra1, Yumna Choudhry2, Stephanie E Coen4,Martha MacLeod5, Christopher Manuel1, Ryan McCarthy2, Adrian Mota2, David Peckham1,Jacqueline M Tetroe2 and Joanne Tucker1AbstractBackground: The Canadian Institutes of Health Research (CIHR) has defined knowledge translation (KT) as adynamic and iterative process that includes the synthesis, dissemination, exchange, and ethically-sound applicationof knowledge to improve the health of Canadians, provide more effective health services and products, andstrengthen the healthcare system. CIHR, the national health research funding agency in Canada, has undertaken toadvance this concept through direct research funding opportunities in KT. Because CIHR is recognized withinCanada and internationally for leading and funding the advancement of KT science and practice, it is essential andtimely to evaluate this intervention, and specifically, these funding opportunities.Design: The study will employ a novel method of participatory, utilization-focused evaluation inspired by theprinciples of integrated KT. It will use a mixed methods approach, drawing on both quantitative and qualitativedata, and will elicit participation from CIHR funded researchers, knowledge users, KT experts, as well as other healthresearch funding agencies. Lines of inquiry will include an international environmental scan, document/datareviews, in-depth interviews, targeted surveys, case studies, and an expert review panel. The study will investigatehow efficiently and effectively the CIHR model of KT funding programs operates, what immediate outcomes thesefunding mechanisms have produced, and what impact these programs have had on the broader state of healthresearch, health research uptake, and health improvement.Discussion: The protocol and results of this evaluation will be of interest to those engaged in the theory, practice,and evaluation of KT. The dissemination of the study protocol and results to both practitioners and theorists willhelp to fill a gap in knowledge in three areas: the role of a public research funding agency in facilitating KT, theoutcomes and impacts KT funding interventions, and how KT can best be evaluated.* Correspondence: robert.mclean@cihr-irsc.gc.ca1Evaluation Unit, Canadian Institutes of Health Research, 160 Elgin Street,Ottawa, ON, CanadaFull list of author information is available at the end of the articleImplementationScience© 2012 McLean et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.McLean et al. Implementation Science 2012, 7:57http://www.implementationscience.com/content/7/1/57BackgroundAround the world and across the spectrum of scientificand non-scientific goods and services, there is a desirefor service and product provision to be informed by evi-dence. This desire has been made explicit within therealm of health research through the concept and idealof evidence-based practice. However, the fact remainsthat health practice often lags behind knowledge andbest practices established through health research [1,2].To address this issue, efforts have been made to promoteevidence-based practice and the use of research in prac-tice. This is a concept that has become known by manynames, including knowledge translation (KT) [3,4].At the Canadian Institutes of Health Research (CIHR),Canada’s national health research funding agency, KT isdefined as a dynamic and iterative process that includessynthesis, dissemination, exchange, and ethically-soundapplication of knowledge to improve the health of Cana-dians, provide more effective health services and pro-ducts, and strengthen the healthcare system [5]. In moresimple terms, KT at CIHR is about having research actas a driver of appropriate real-world applications. KThas been an important aspect of CIHR’s vision and worksince the organization’s inception in 2000. In fact, toformalize the importance of KT to the organization, itwas embedded in the CIHR mandate and written intothe Parliamentary act that created CIHR as it now exists[6]. What this means in practice is that CIHR has writ-ten KT into its strategic plan, with a directive to acceler-ate the capture of the benefits of health research; createdexecutive management roles and a unique branch of theorganization devoted expressly to KT; and developedKT-specific funding mechanisms—the focus of thepresent evaluation protocol. The range of CIHR’s stra-tegic activities and funding opportunities to address KTare designed to support not only KT science but also allof the elements of CIHR’s definition of KT (synthesis,dissemination, exchange, and ethically sound applicationof knowledge).Paradoxically, and we believe to its disadvantage, thefield of KT has lagged in what it is designed to address—the use of evidence to inform better products, services,and systems. Despite the fact that much evidence existsto support the need for KT, very little evidence existsthat measures the performance and impact of KT inter-ventions, especially when those interventions are fund-ing mechanisms [4,7].The evaluation research described in this protocol isdesigned to address this shortcoming of concern to prac-titioners and theorists alike. The aim of this study proto-col, and the dissemination of its subsequent results, is togenerate evidence in relation to the role of a public re-search funding agency in enabling/promoting KT, theoutcomes and impacts of KT funding interventions, andhow KT can best be evaluated. The study will investigatehow efficiently and effectively the CIHR model of KTfunding programs operates, what immediate outcomesthese funding mechanisms have produced, and what im-pact these programs have had on the broader state ofhealth research, health research uptake, and health im-provement. The need for further research on the effect-iveness of KT is especially imminent for the publicfunding agency, where KT interventions are designed tobenefit the whole of society and are financed to do so bythe taxpayer.As very little evaluation has been conducted on theperformance and promotion of KT, the study describedin this protocol represents a unique approach to thiscomplex task. The approach is grounded in the theoret-ical frameworks of both evaluation and KT. The remain-der of the protocol presents an overview of thisundertaking. The section that follows describes the ap-proach to scoping an evaluation of CIHR KT interven-tions. The remaining sections outline the studymethodology and provide a discussion of potential studyimplications.Scoping the studyEvaluation purposePrimarily, this evaluation study is designed to provide validand insightful findings about the performance of CIHR’sKT programs for the purposes of program learning andfuture KT program development. The study will investi-gate how efficiently and effectively the CIHR KT fundingprograms operate, what immediate outcomes these fund-ing mechanisms have produced, and what impact theseprograms have had on the broader state of health research,health research uptake, and health improvement.The evaluation is also designed to meet CIHR’srequirements to Canada’s Treasury Board Secretariat(TBS) in order to demonstrate value for money in gov-ernment spending. It therefore covers specific core TBSevaluation issues of program relevance and performanceas described in the TBS policy suite a. In the discussionsection, we elaborate on the implications of designingthe evaluation protocol to meet both our prospectiveprogram learning and development objectives, and retro-spective accountability and reporting objectives.The CIHR Act (Bill C-13) mandates CIHR to ensurethat the translation of health knowledge permeates everyaspect of its work [6]. An evaluation of all knowledgetranslation programs and activities at CIHR would there-fore need to be broad in scope, be extremely resourceintensive, and as such would likely only be able to pro-vide very high-level findings. The intent of this evalu-ation is to provide evidence about the performance ofCIHR’s overall KT strategy, but also to provide moredetailed findings about the intricate factors surroundingMcLean et al. Implementation Science 2012, 7:57 Page 2 of 16http://www.implementationscience.com/content/7/1/57the design and delivery of this strategy (i.e., the targetedfunding opportunities). Accordingly, the study outlinedin this protocol is designed to address some of the keyconstraints and limitations of evaluating KT at CIHRincluding:1. Ensuring that the evaluation is sufficiently targetedto investigate the idiosyncratic factors surroundingindividual KT funding opportunities;2. Ensuring the evaluation gathers data that reflect theoperational definition of KT at CIHR (synthesis,dissemination, and ethically sound application ofknowledge);3. Ensuring that the evaluation is designed to gatherdata required by the Treasury Board while alsocollecting data that will address our objectivesrelated to program learning and development;4. Performing such research with a limited set ofresources (particularly, time and financialconstraints).To address these considerations, a purposive samplingapproach to selecting a set of KT funding programs wasdeveloped. Focusing on a comprehensive sample offunding programs enables the evaluation to providedetailed, precise, and useful findings at the programlevel. Triangulated and rolled-up, this data will providean indication of the overall performance of CIHR’s KTstrategy. Five CIHR KT funding domains were selectedto be within the scope of this evaluation:1. Knowledge Synthesis funding opportunity;2. Partnerships for Health Systems Improvement(PHSI) funding opportunity;3. Knowledge to Action (K2A) funding opportunity;4. End of grant KT funding opportunities(Dissemination Events (DE) and the KT Supplement(KTS) programs)b;5. KT research funding opportunityc.These programs were selected and validated for inclu-sion based on two key criteria—program relevance andprogram materialityProgram relevancePrograms were selected in order to provide full theoret-ical coverage of the four fundamental KT themes identi-fied by CIHR management as representative of thepurpose and concept of KT at CIHR. Details of these areprovided in Table 1. The relevance review of KT pro-grams was conducted through formal consultation withCIHR senior management and KT specific staff, and vali-dated by external expert opinion d.Program materialityCIHR financial records were reviewed to assess the ma-teriality of each KT program, and a risk-based approachto selecting programs was applied. The five selecteddomains represent approximately 65% of current finan-cial commitments for the KT area at CIHR e,f. Excludingthe KT research funding opportunity within the openoperating grants program (OOGP), the four programsalso represent the four largest individual financial com-mitments provided through KT funding at CIHR.KT at CIHR and specific KT funding programs beingexaminedIn this section, contextual details are provided on theconcept of knowledge translation and the strategic ap-proach to KT at CIHR. Subsequently, a description isprovided of the funding programs that will be specific-ally examined through the evaluation study.Figure 1 is a logic model g that was produced as a partof the planning of this evaluation. It provides a visualrepresentation of the object of the evaluation.The concept of knowledge translation at CIHRTo promote/enable the concept of KT as defined byCIHR, there are four aspects supported by theorganization: knowledge synthesis, integrated KT, end ofgrant KT, and KT science.Table 1 Relevance and materiality coverage of selectedKT funding programsSelectedfundingprogramKT area offocusCAD(mil) - 2010-11financialcommitment% of 2010-11 KTfinancialcommitmentKnowledgeSynthesisSynthesis;IntegratedKT1.76m CAD 10.6%Partnershipsfor HealthIntegratedKT6.29m CAD 37.7%SystemImprovementKnowledgeto ActionIntegratedKT1.33m CAD 8%DE and KTS End grantKT1.34m CAD 8%KT research KT science n/a n/aTotal coverage 10.72m CAD 64.3%Notes: 1) Figures are based on finance coding for CIHR’s PAA 1.4.2.; 2) KTresearch financial data is not included for the current period as money is notmoved directly through PAA 1.4.2.; 3) Programs not included in our evaluationas a whole represent 35.7% of PAA area spending (less Partnerships programsthat were purposely removed) and for this period are: Reduce HealthDisparities, Training Awards, CADRE, Clinical Research Initiatives, HealthResearch Community Awards, KT Awards, Mobility in Aging, Cochrane Canada,Youth and Public Engagement, Res Action Program in Dementia, Partnershipsaward, JBI, Journalism Awards, Evidence Review and Synthesis Centre,Canadian Knowledge Synthesis Network, Canadian Virtual Library Network.McLean et al. Implementation Science 2012, 7:57 Page 3 of 16http://www.implementationscience.com/content/7/1/57Synthesis is the contextualization and integration ofresearch findings of individual research studies withinthe larger body of knowledge on the topic. It is a familyof methodologies developed to determine what is knownin a given area or field and what the knowledge gaps are.The underlying principle is the support of timely and ac-curate scientific knowledge being available to those whowork in real-world settings requiring such evidence.Knowledge synthesis studies may be useful to policy-makers, industry, clinical, and medical practitioners,amongst others. In some cases, knowledge synthesis re-search can be conducted with the participation of non-traditional researchers throughout the research process.CIHR refers to the engagement of knowledge users inresearch as integrated KT (iKT).Through iKT, stakeholders or potential knowledgeusers are engaged in the entire research process. Bydoing iKT, researchers and knowledge users work to-gether to shape the research process by collaborating todetermine the research questions, deciding on themethodology, being involved in data collection andtools development, interpreting the findings, and help-ing disseminate the research results. This approach isdesigned to produce research findings that are morelikely to be relevant to and used by end users. This ap-proach is similar to those described as collaborative re-search, participatory, action-oriented research, co-production of knowledge, and Mode 2 knowledgeproduction.End of grant KT describes the process where the re-searcher develops and implements a plan for makingknowledge users aware of the knowledge that was gainedduring a project. End of grant KT includes the typicaldissemination and communication activities undertakenby most researchers, such as KT to their peers throughconference presentations and publications in peer-reviewed journals. End of grant KT can also involvemore intensive dissemination activities that tailor theFigure 1 Knowledge translation funding programs logic model.McLean et al. Implementation Science 2012, 7:57 Page 4 of 16http://www.implementationscience.com/content/7/1/57message and medium to a specific audience, such assummary briefings to stakeholders, interactive educa-tional sessions with patients, practitioners, and/or policymakers, media engagement, or the use of knowledgebrokers to name a few. The commercialization of scien-tific discoveries is another form of end of grant KT, butas a specific strategy it is not being explored in thisstudy.KT science or research (also known as implementationscience) is the study of the process of KT and the use ofknowledge. KT science explores the factors that facilitateand hinder the sharing of knowledge between creatorsand users. While it often addresses issues such as the ef-ficacy of certain KT strategies, KT science may also in-volve the development of new KT theory or practice.Specific KT funding programsKnowledge synthesis funding opportunityKnowledge synthesis grants provide funding to research-ers to produce scoping reviews or syntheses that meetthe needs of decision makers or knowledge users in allareas of health. They support the concept that know-ledge users should identify synthesis questions in collab-oration with researchers so that the answers to thesequestions can inform policy, programs, and practice.They are also expected to increase the capacity ofresearchers to identify new, relevant avenues for explor-ation that have not yet been investigated that respond todecision makers’ and knowledge users’ needs [8]. Finally,because they are funded to be performed in an iKT for-mat, synthesis grants are intended to promote theprocess of mutual learning between researchers andknowledge users.First launched in 2004, CIHR invites all forms ofknowledge synthesis. Qualitative, quantitative, andmixed methods approaches are accepted, as well as syn-theses of knowledge gained through observation, testing,or reviewing of texts. Scoping reviews are also eligible;these are projects that explore the literature available ona topic, identifying the key concepts, theories, sources ofevidence, and gaps in the research. They are oftenundertaken before a full synthesis when the literature isthought to be too vast or when there is suspicion thatnot enough literature exists to synthesize [8]. Becausethe knowledge synthesis funding opportunity is fundedas iKT, applications to the funding opportunity undergoa merit review process.Merit review is markedly different than typical CIHRpeer review. The composition of iKT research teamsand/or the nature of KT research projects require thatmerit review panels expand the traditional definition of‘peer’ to include knowledge users whose expertise lies inthe application of research. Because both researchersand knowledge users contribute to the production andthe translation of research, merit review panel compos-ition must reflect this, drawing members from both re-searcher and knowledge user communities. Eachapplication is reviewed by at least one researcher and atleast one knowledge user who assess potential impactand scientific merit; potential impact and scientific meritare weighted equally. Only those applications receiving afundable score on both potential impact and scientificmerit can be considered for CIHR funding [9].ResourcesKnowledge synthesis competitions are launched twice ayear by the CIHR Knowledge Translation Branch inpartnership with various CIHR institutes and strategicinitiatives, along with external partners. The maximumamount awarded for a synthesis is 100,000 CAD for oneyear. The maximum amount awarded for scopingreviews is 50,000 CAD for one year.Partnerships for health system improvement (PHSI)funding opportunityThe first CIHR PHSI competition was held in 2005 afterit was transferred from Canadian Health Services Re-search Foundation. The PHSI funding program supportsteams of researchers and decision makers/knowledgeusers interested in conducting applied and policy-relevant health systems and services research that re-spond to the needs of healthcare decision makers. Part-nerships can be project specific (partners that theresearchers identify themselves) or competition specific(CIHR negotiated competition partnerships). This fund-ing opportunity requires pre-defined financial or in kindpartner contributions [10]. PHSI grants are funded to beperformed in an iKT format, and as such, this programuses a merit review process to evaluate applications.ResourcesThe maximum amount awarded by CIHR for a singlegrant is 400,000 CAD for up to three years (partnershipcontributions are in addition to the CIHR amount). Aminimum of either 20% or 30%, depending on the prov-ince or territory, of the grant budget must come fromexternal partner sources (i.e., non-CIHR funds). There isno limit to partner contributions, and in-kind contribu-tions are recognized, especially where they reflect mean-ingful collaboration that will increase the likely successof the project. It should be noted that funding and con-tributions may be received from stakeholders who arenot members of the grant team.Knowledge to action (K2A) funding opportunityK2A is designed to move knowledge into action by link-ing researchers and knowledge users and to increase theunderstanding of knowledge application through theMcLean et al. Implementation Science 2012, 7:57 Page 5 of 16http://www.implementationscience.com/content/7/1/57process. By bringing both parties together, it is expectedthat research results will translate to actions thatstrengthen Canada's healthcare system and/or improvethe health of Canadians. K2A also aims to support thedevelopment, implementation, and evaluation of cutting-edge KT research and approaches. Through this, theprogram establishes and strengthens common groundbetween the interests and expertise of the research com-munity and the needs of knowledge users. Applicantscan request funding to support partnerships, knowledge,and tools for implementation projects. This programwas first launched in 2005. Because the K2A funding op-portunity requires iKT, all applications go through amerit (not peer) review process.ResourcesThe maximum CIHR contribution is 100,000 CAD peryear for up to two years. Applicants may increase fund-ing for their proposal and further demonstrate the levelof engagement of their partner(s) through cash or in-kind commitments, but a financial commitment fromthe partner is not a criterion for funding [11]. Applicantsare encouraged to apply for a renewal of their grant ifthey plan to scale up their implementation project.End of grant KT funding opportunities: Disseminationevents (DE) and the KT supplement (KTS)The DE and KTS funding opportunities both supportend of grant KT. DE is intended to provide support formeetings, and/or dissemination activities consistent withthe mandate of CIHR and relevant CIHR institutes,initiatives, or branches. It supports the organization ofevents focused on the communication of health researchevidence. The KT Supplement funding opportunity sup-ports KT activities that follow implementation of a peer-reviewed grant/award where further dissemination is ap-propriate. Both DE and KTS applications undergo a peerreview, rather than merit review, process.Eligible activities for the DE funding opportunity include:1. Education of groups such as patients, healthprofessionals, community organizations, policy-makers, the general public;2. Education of stakeholders regarding partnership bestpractices;3. Knowledge dissemination that will inform practice,clinical care, policy and decision making;4. Publishing articles in open access journals notbudgeted for in other applications, as part of abroader dissemination strategy.Eligible activities for the KTS funding opportunityinclude:1. Development/maintenance/updating of websites;2. Production and distribution of written materials invarious formats;3. Hiring of a knowledge broker or implementationfacilitator/change agent;4. Development of plain language summaries;5. Development of knowledge exchange tools (e.g.,educational DVDs, decision support tools);6. Dissemination of research results throughspecialized publications as part of a broader KTstrategy, and;7. Travel costs for a series of meetings/presentations(linkage and exchange activities) required toimplement a broader KT strategy.ResourcesThese end of grant KT funding opportunities are non-renewable one-year grants. However, multiple grants canbe awarded to the same candidate in the same calendaryear. DE projects are funded up to 25,000 CAD, whileKTS projects are funded up to 100,000 CAD [12].KT research funding opportunity within the OOGPFunded KT research grants must be directed towarddeveloping theory, evidence, and innovation to definethe determinants, implementation, and uptake of healthresearch evidence into practice. These include grantsthat aim to improve KT to consumers, health practi-tioners, and policy makers, to examine the role of orga-nizations as KT vehicles, to determine how to improveknowledge uptake potential during the research process,to develop/evaluate KT tools and/or methods, and tocontribute to KT theory and to improve knowledge up-take. KT research grants do not require knowledge userpartners, although they are allowed, and are peer (notmerit) reviewed.ResourcesThese OOGP competitions provide funding for up tofive years and have no funding limit or specific require-ments for team size or composition. Funding is allocatedthrough the CIHR open operating grant budget ratherthan CIHR’s KT-specific budget. The Knowledge Trans-lation Research committee is one of 53 standing com-mittees on the OOGP.MethodsThe following section outlines the investigation processthat will be employed in this research. Each method ofinquiry is described, and a brief preface about theprocess of design is included.McLean et al. Implementation Science 2012, 7:57 Page 6 of 16http://www.implementationscience.com/content/7/1/57Design and processThe study will employ a novel method of participatory,utilization-focused evaluation inspired by the notion ofiKT. The study will use a mixed methods approach,drawing on both quantitative and qualitative data, andwill elicit participation from stakeholder groups, includ-ing CIHR funded researchers, knowledge users, KTexperts, TBS, and other health research funding agen-cies. The use of a mixed methods approach will be bene-ficial to uncovering significant detail about this complexintervention [13].Utilization-focused evaluation is based on the idea thatevaluations are only as efficacious as they are useful totheir consumer(s). Patton [14] describes utilization-focused evaluation as being established on the premisethat evaluations should be judged based on their actualuse, and therefore, from planning to conclusion theyshould be conducted in the manner that is best adaptedfor intended end users. To us, stakeholders are muchmore likely to use this evaluation if they feel ownershipover the evaluation purpose, process, and findings. Byactively involving users throughout this evaluation, thefoothold for use is being established and the utility ofthe evaluation is being continuously reinforced.To realize this utilization approach, we have designedthe protocol through a collaborative approach betweenmultiple stakeholders, assembled in a research team thatwe have called the Evaluation Working Group. TheEvaluation Working Group is chaired by CIHR’s Evalu-ation Unit, and included a broad spectrum of CIHR staffinvolved in developing, delivering, and evaluating theprograms; these members represented key internal KTstakeholders with operational knowledge of KT programdesign and delivery (representing the KT Branch as wellas a CIHR institute). The Evaluation Working Groupalso included an external (non-CIHR employed) re-searcher with KT expertise and funding who also servesas the chair of one of the merit review panels of interest.The combination of CIHR internal and external perspec-tives on the Evaluation Working Group ensured that theprotocol development was grounded in the operationalrealities of CIHR and designed to provide appropriateinput for program improvement purposes, while beingattuned to practicalities of engaging with these programsand implementing funded projects on the program userside. The intent is to conduct the evaluation in a partici-patory fashion similar to what CIHR expects of appli-cants to its iKT funding programs. The EvaluationWorking Group members involved in the design phasewill remain involved through the entire research process.Indeed, conducting this research in a collaborative fash-ion will facilitate the utility of the evaluation throughboth process and product (or findings) benefits, and, assuch, support the use of the evaluation [15]. Ad hocparticipation from each group member is to be expectedand encouraged, however, at five critical stages the entiregroup will meet to seek consensus and affirm their satis-faction and that their representation is upheld:1. The design of the evaluation framework—thisincludes the study sampling process (describedabove), the design of the logic model, the design ofresearch questions, and the selection of methods.2. At the data collection phase the team will review allinstruments and processes and take part incollection where appropriate.3. When data are collected, the team will reviewfindings from individual perspectives and then meetto form a group consensus on final interpretationand to learn how others reflections complement ordetract from their own.4. After measured contemplation of the findings, thegroup will consult to discuss best methods ofdeveloping an action plan to implement evaluationrecommendations.5. After measured contemplation of the findings, thegroup will consult to discuss best methods ofdissemination to stakeholders, both external andinternal.Evaluation questionsThe evaluation will be focused on addressing a set ofoverarching questions regarding CIHR KT strategythrough investigating funding program performance. Inorder to maximize the utility of the evaluation, thesequestions were developed collaboratively with the Evalu-ation Working Group.The questions provide the overall direction for theevaluation; a series of detailed indicators and datasources designed to address these has been developed.The overall evaluation questions are as follows:1. What role is there for CIHR in enabling/promotingiKT research, synthesis, end of grant KT, and KTscience?2. To what extent are KT funding programs achievingtheir expected outcomes?3. What factors facilitate or inhibit the achievement offunding program outcomes?4. How effective is the mix of KT funding programs inachieving CIHR’s expected outcomes? (iKT, end ofgrant KT, KT science, synthesis)5. To what extent have KT funding programs reacheda broad and diverse range of knowledge users?6. To what extent are KT funding programs beingdelivered as expected? Can any changes be made toprogram delivery in order to improve efficiency andeffectiveness?McLean et al. Implementation Science 2012, 7:57 Page 7 of 16http://www.implementationscience.com/content/7/1/577. What would be the effect on CIHR-fundedresearchers and knowledge users if the KT programsuite no longer existed? What would be the effect onthe improvement of health, more effective healthservices and products, and the strengthening of thehealthcare system?8. What are the unanticipated outcomes, positive ornegative, resulting from the KT fundingprograms?The Evaluation Matrix in Table 2 provides full detailsof indicators and data sourcesTable 2 Evaluation MatrixEvaluation questions Indicators Methods Sources1. What role is there for CIHR in enabling/promotingsynthesis, iKT, end-of-grant KT, and KT science?● Theory and empirical evidence related to the roleof a funding organization in the KT process■ Internationalenvironmentalscan■ 33 funding agenciesfrom Tetroe et al.2008 study● Is the CIHR role consistent with the health needs ofCanadians, the improvement of health products andservices, and the strengthening of the Canadianhealthcare system?● Theory and empirical evidence related to theadvantages and limitations of iKT research, end-of-grant KT, and KT science● Degree of alignment of CIHR KT funding programsuite with theory and empirical evidence of KTsuccess strategies● Organizational scan of comparable organizationsnationally and internationally● Expert opinion on the role of a fundingorganization in the KT process■ Externalexpert review■ International KTexpert panel● Expert opinion on the CIHR funding program mix● Expert opinion on CIHR strengths, limitations, andstrategic vision for KT funding programs● Indications of incentive induced behaviour ofresearchers and knowledge users■ Case studies ■ Exceptional fundedprojects● Indications of unique or innovative KT strategiesemployed● Application pressure (total applications perfunding program)■ Documentand EIS datareview■ EIS applicationrecords● Ratio of researchers funded versus applied ■ CIHR guidingdocuments● Ratio of researchers funded versus fundable butnot funded■ Government of● Degree of alignment with CIHR mandate andstrategic vision■ Canadadocumentation● Degree of alignment with the government ofCanada’s plans and priorities? (i.e. SandT Strategy)2.To what extent are KT funding programs achievingtheir expected outcomes?● Indications of immediate, intermediate, and long-term outcomes■ Surveys ■ Funded researchers/knowledge users● To what extent are immediate outcomes beingachieved?● To what extent are intermediate outcomes beingachieved?■ Keyinformantinterviews■ Funded researchers/knowledge users■ Case studies ■ Exceptional fundedprojects● The number of grants awarded by each program ■ Documentand EIS datareview■ EIS applicationrecords● # of partnerships created (iKT) ■ End of grant reportsMcLean et al. Implementation Science 2012, 7:57 Page 8 of 16http://www.implementationscience.com/content/7/1/57Table 2 Evaluation Matrix (Continued)● Comparison of application pressure acrossfunding programs● Indications of intermediate and long termoutcomes● Degree of alignment of KT funding progam suitewith theory and empirical evidence of KT successstrategies■ Internationalenvironmentalscan■ 33 funding agenciesfrom Tetroe et al.2008 study■ Externalexpert review■ International KTexpert panel3. What factors facilitate or inhibit the achievement ofoutcomes?● Indication of influence on program theory from::internal program processes; external environmentalfactors; strategic level factors; program delivery levelfactors■ Surveys ■ Funded researchers/knowledge users■ Keyinformantinterviews■ Funded researchers/knowledge users■ Case studies ■ Exceptional fundedKT projects■ Documentand EIS datareview■ EIS applicationrecords■ Final reports● Program delivery level factors4. How effective is the mix of funding programs inachieving CIHR’s expected outcomes? (iKT, End ofgrant-KT, KT Science, Synthesis)● Perceptions of suitability of program mix forpromoting/enabling effective KT■ Keyinformantinterviews■ Funded researchers/knowledgeusers■ Surveys ■ Funded researchers/knowledge users● Profiles of pathways to program outcomes ■ Case studies ■ Exceptional fundedKT projects● Degree of alignment of CIHR KT funding programsuite with theory and empirical evidence of KTsuccess strategies■ Externalexpert review■ International KTexpert Panel■ Internationalenvironmentalscan■ 33 funding agenciesfrom Tetroe et al.2008 study5. To what extent have KT funding programs reacheda broad and diverse range of knowledge users?● Number and type of knowledge users includedper iKT grant■ Documentand EIS datareview■ EIS applicationrecords■ Final reports● Perceptions of meaningful partnerships havingbeen established■ Surveys ■ Funded researchers/knowledge users■ Keyinformantinterviews■ Funded researchers/knowledge users■ Case studies ■ Exceptional fundedKT projects6. To what extent are KT funding programs beingdelivered as expected? Can any changes be made toprogram delivery in order to improve efficiency andeffectiveness?● Indications of efficiency and effectiveness in theconversion of program activities into programoutputs■ Documentand EIS datareview■ EIS applicationrecords■ Surveys ■ Funded researchers/knowledge● Identified success and challenges of the meritreview process■ users■ Keyinformantinterviews■ Funded researchers/knowledge usersMcLean et al. Implementation Science 2012, 7:57 Page 9 of 16http://www.implementationscience.com/content/7/1/57Methods of investigationTo ensure that findings are robust and that valid conclu-sions can be drawn about the performance of the pro-grams, the evaluation will use multiple methodologiesand draw on both quantitative and qualitative evidence.A range of methods will be employed to capture a widediversity of data, namely: an international environmentalscan of health research funding agencies; a document re-view and CIHR’s Electronic Information System (EIS)data review; in-depth key informant interviews; quantita-tive surveys; case studies; and an external expert paneldiscussion. A range of quantitative and qualitative dataanalysis techniques will be used to interpret each sourceof data, and are described under each heading below. Toensure rigour, our analysis will triangulate findings fromall methods to inform study conclusions.International environmental scanA review of organizations from a range of countries thatprovide KT research funding will be conducted in orderto gather information regarding the how they fund KTand what might be considered best practices in the field.The environmental scan will focus predominately on thefirst evaluation question (CIHR’s role in enabling/pro-moting KT).The scan will be formulated as an update and expan-sion on a 2008 publication by Tetroe et al. entitled‘Health research funding agencies' support and promo-tion of knowledge translation: an international study.’The scan will use the same sample frame of organiza-tions. A review of each agency’s website and publicdocumentation will be performed, and follow-up semistructured interviews with a KT contact person and anTable 2 Evaluation Matrix (Continued)■ Case studies ■ Exceptional fundedKT projects7. What would be the effect on CIHR-fundedresearchers and knowledge users if the KT fundingprogram suite no longer existed? What would be theeffect on the improvement of health, more effectivehealth services and products, and the strengtheningof the healthcare system?● Perceived impact of absence of future KT fundingon funded researchers, knowledge users, and KToutcomes■ Surveys ■ Funded researchers/knowledge users● Perceived future directions for funded researchers,knowledge users, and KT outcomes in the absenceof KT funding■ Keyinformantinterviews■ Funded researchers/knowledge users● Use of alternative funding sources by KT fundedteams (leveraging)■ Case studies ■ Exceptional fundedKT projects● Use of alternative funding sources by KTresearchers and knowledge users not funded byCIHR (Knowledge User partners)■ EIS ■ EIS applicationrecords● Organizational scan of similar organizationsnationally and internationally■ Final reports■ Internationalenvironmentalscan■ 33 funding agenciesfrom Tetroe et al.2008 study8. What are the unanticipated outcomes, positive ornegative, resulting from the KT funding programs?● Identified unintended outcomes of KT fundingprograms■ Documentand EIS datareview■ EIS applicationrecords■ Final reports■ Surveys ■ Funded researchers/knowledge users■ Keyinformantinterviews■ Funded researchers/knowledge users■ Case studies ■ Exceptional KTfunded projects■ International ■ 33 funding agencies■environmentalscan■ from Tetroe et al.2008 study■ Externalexpert review■ International KTexpert panelNote: Indicators and sources presented in this matrix are not static. As the research process progresses, the Evaluation Working Group will be attuned to newinformation that may create the need for reviewMcLean et al. Implementation Science 2012, 7:57 Page 10 of 16http://www.implementationscience.com/content/7/1/57evaluation contact person from each agency will be pur-sued. Completed data templates will be sent to eachorganization for validation.The environmental scan will provide context and evi-dence surrounding the role of a funding agency in KTprocesses, the known successes and limitations of vari-ous KT funding programs, as well as KT evaluation. Acomprehensive review of empirical evidence related tothese three subjects will allow for the development of acontextual base for the remainder of data collectionphases, and will situate CIHR in comparison to similarorganizations around the world. This component of theevaluation will also yield important insights that willcontribute to global literature on KT science, specificallyfilling a knowledge gap in relation to public fundinginterventions for KT and their evaluation. The Evalu-ation Working Group will be engaged throughout theprocess.Document review and electronic information system datareviewA document and data review will be a significant sourceof information for this study in order to address each ofthe evaluation questions. Documentation to be reviewedwill include key CIHR publications and Government ofCanada publications related to the study topic. TheEvaluation Working Group will work closely to identifyand locate key documentation that is pertinent to eachstakeholder group (i.e., CIHR staff, KT researchers andknowledge users, program users, TBS).CIHR’s EIS data will be used to obtain and analyze ap-plicable information concerning KT funding programapplicants. Where necessary, the CIHR Finance Unit willbe approached for financial datasets.Although initial data and document mining will bebased on identified questions and indicators, the docu-ment and EIS data review will be ongoing throughoutthe data collection phase. As such, the process will bereactive to the most current discoveries and suggestionsfrom other lines of investigation and provide an ongoingsource for triangulation of findings.Key informant interviewsSemi-structured in-depth interviews with researchersand knowledge users awarded CIHR KT grants will beconducted to gather information on key stakeholders’perceptions and experiences with CIHR KT funding pro-grams. The qualitative data gathered through the inter-views will provide important context to issues exploredthrough surveys and data review. Qualitative data are,for example, particularly useful in understanding whyparticipants hold particular views, or when seeking tounderstand a more complex interaction or procedure.Interviews will be conducted with two discrete samplegroups, in order to capture a diverse and balanced viewof performance. Researchers and knowledge users willbe interviewed, where possible from the same fundedKT project. The combination of the two perspectivesfrom a single project will be used to elicit the sharedand distinct opinions of the two. It is anticipated thatthis approach will unearth robust detail. Up to 30 inter-views will be conducted, or until saturation is reached,with funded researchers (n 15 interviews) and fundedknowledge users (n 15 interviews).Interviewees will be selected on the basis of the follow-ing criteria: funding program used, experience withCIHR/CIHR KT research, research area (CIHR researchpillar i.e., biomedical, clinical, health services, social, cul-tural, environmental, and population), Canadian officiallanguage (French or English), and geographic location.The interviews will be conducted by telephone andversioned interview guides will be developed for the re-searcher and knowledge user interviews. The tailoring ofguides to each stakeholder group will illuminate differingexperiences and perspectives. Interviews will be designedto be approximately 45 to 90 minutes in length. Allinterviewees will be afforded full confidentiality in theirresponses, and collected notes and recordings will bemanaged in accordance with the federal Privacy Act.Interview data will be coded and analysed using NVivosoftware; data will also undergo review by the EvaluationWorking Group to identify and recount key subjectiveexperiences of the interviewees using constant compara-tive analysis (i.e., taking data and comparing it to othersthat may be similar or different).Quantitative surveyA quantitative survey will be used in order to gathermore generalizable information related to funding pro-gram performance. The quantitative survey will belaunched following the key informant interviews. Thisstructure will allow for specific lines of investigation inthe survey to be informed by interview responses. Fur-thermore, it allows for a test of language and questionframing issues to be performed in the interactive inter-view setting, and thus, for the survey to be framed in themost appropriate way for the target population. The sur-vey design was developed through an iterative feedbackprocess with the Evaluation Working Group.Funded researchers will be surveyed, as well as a coun-terfactual group h of researchers funded through CIHR’snon-KT funding opportunities. The partial coverage of theCIHR KT funding programs (i.e., not everyone who is eli-gible applies to the programs) allows for comparison be-tween a group of KT funding program applicants andnon-applicants, and will help identify program effects andimpact.McLean et al. Implementation Science 2012, 7:57 Page 11 of 16http://www.implementationscience.com/content/7/1/57Surveys will be administered to the full population ofrecipients of each of the five KT funding programs (n 600). The comparison (counterfactual) group will beselected based on responses to CIHR’s research report-ing instrument, which was recently piloted. This isCIHR’s general end of grant reporting tool, and it hasbeen administered to recipients of open operating grants(not these KT programs).The survey will be hosted online, and participants willbe invited to take part by email. In order to minimizeburden on respondents, surveys will be designed to takeless than 30 minutes to complete. Questionnaires will bedeveloped with reference to overarching evaluationquestions and will be tailored to each funding program,but will also have commonalities to allow for valid com-parisons across groups. Respondents will be assured offull confidentiality. The survey will be undergo a pre-testperiod to allow for corrections and streamlining.Where appropriate, we will draw on the design ofquestionnaires used in previous evaluations of KT pro-grams, both within CIHR and externally. Data collectedfrom surveys will be analysed appropriately (i.e., bivariateand multivariate cross examination where statisticallyrobust) using SPSS. Results will be reviewed and inter-preted by the Evaluation Working Group.Case studiesA total of five case studies will be undertaken in orderto investigate and illustrate links between funding pro-gram activities and outcomes. A case study method ofinquiry will provide empirical data regarding occur-rences of KT and a frame of context surrounding thesetting where this trend and/or occurrence took place.The case studies will employ a pathway case analysis for-mat. This analysis method is useful when the result ofan intervention is known (in our case, an exceptionaldemonstration of expected KT outcomes), and the start-ing point of the intervention is relatively similar (in ourcase, a funded KT project). The pathway analysis allowsfor the investigation of causal factors of influence affect-ing the intervention [16]. Case study investigation willsupplement other lines of evidence by providing richand detailed accounts of the knowledge translationprocess.One case study will be conducted for each of the fiveKT funding programs. The selection of one successfulfunded project per program will be an interactiveprocess engaging the Evaluation Working Group Pro-jects will be selected that demonstrate exceptionalinstances of KT outcomes, so that lessons can be drawnabout what pathway factors lead to success.Case studies will be developed based on a commonsemi-structured interview process with a funded team ofresearchers and knowledge users, a review of projectdocumentation, and site visits where appropriate. Acommon approach to data collection will allow for theanalysis of similar issues and questions across variedprojects and the meaningful comparison of findings. Theapproach will be developed through consultations of theEvaluation Working Group, and case study drafts will bereviewed by all group members.With this in mind, the design of the case study re-search will not be a ‘checklist’ approach built againstpre-determined indicators. The approach will provideample flexibility for the documentation of not only theKT process within a project, but equally important, theenvironment in which the process occurred. Document-ing this environment will provide valuable context to theKT processes and to understanding their success.External expert review panel discussionAn expert review will be undertaken in order to provideexpert insight into the CIHR KT funding programs and toprovide an arm’s length assessment of the evaluation andits findings. The perceived position of CIHR as a Canadianand global leader in KT provides a unique opportunity forattracting the interest of leading subject area experts toprovide advice and opinion on CIHR funding programstrategy. At the same time, this position of leadershipnecessitates critical review by the most accomplished ofspecialists. Reviewers invited to participate the panel willbe KT specialists of international repute, the majoritybeing from countries other than Canada; some Canadianexperts who have received funding from CIHR may beincluded. No CIHR staff will be on the expert panel inorder to reduce bias in interpreting the data.The study will be designed to provide a forum for dis-cussion between leading KT area experts. The panel willalso review the data, analysis and interpretations of theEvaluation Working Group, and be asked to commenton the rigour and accuracy of the evaluation. More pre-cisely, our deliberative approach will involve an expertgroup, a series of iterations where information is col-lected, processed by a moderator, and returned to theEvaluation Working Group members for further analysisbased on collective input. The process will allow forinferences to be drawn by leading thinkers in the field.Issues explored in the study will relate to CIHR KTfunding programs as well as the wider CIHR KT strat-egy. Primarily, evaluation questions one and two will bethe major focus of this line of investigation. However,the study will be timed to conclude the data collectionphase so that key issues arising from each line of evi-dence can be explored in greater depth through themethod.Data collection sessions and communications will bemoderated by the Evaluation Working Group in order toensure neutrality. In order to ensure meaningful results,McLean et al. Implementation Science 2012, 7:57 Page 12 of 16http://www.implementationscience.com/content/7/1/57the Evaluation Working Group will also be fully engagedin the design of the instrument and the selection ofparticipants.Analytic approach: Triangulating data from multiplesourcesThe five methods of data collection described above fittogether as part of a data triangulation strategy. Thecomponents were designed and will be sequentially car-ried out in order to iteratively influence the design ofsubsequent components; in this way, the protocol cumu-latively builds on each data type. These multiple sourcesof data will serve to uphold rigour in our analysis be-cause findings from each component will be cross-checked for consistency and investigated where discrep-ancies arise.Reporting approachThe study protocol (this document), is the first major re-port stemming from the planned evaluation. The pur-pose of this report is to encourage the process of thisevaluation to be shared and criticized, and thus, to en-courage learning about best practices in the evaluationof KT.Final results of the study will be reported on in aggre-gate form in a final evaluation report. Once finalized,this evaluation report will be submitted to TBS andmade publicly available on CIHR`s website.Additional publications, presentations, and other dis-semination items/events will be crafted wherever pos-sible and appropriate, and may be prepared for any ofthe data collection methods individually or in combin-ation. Given that a key driver of this research is the lackof knowledge surrounding KT effectiveness, optimalapproaches to funding KT, and the evaluation of KT,reporting on the process and results of this studybecomes an essential purpose. We also plan to write apaper on our experience conducting the evaluation inthis way.Ethical considerationsThe project is being performed under the auspices ofCIHR’s requirement to evaluate its expenditures as ismandated for all public organizations in Canada’s Treas-ury Board Secretariat Policy on Evaluation. As a part ofour relationship with TBS, the research undertaken inthe evaluation of federal public expenditures is ethicallyauthorized under the Values and Ethics Code for thePublic Service and the Privacy Act. Even as a federalfunding agency responsible for the development and on-going management of the Tri-Agency Policy on EthicalResearch our evaluation research is ethically authorizedunder the aforementioned federal policy, code, and actwithout Research Ethics Board review.However, the study has also been approved by the Ot-tawa Hospital Research Ethics Committee. Full measureswill be taken to uphold research ethics in accordance toour relationship with TBS and the Ottawa Hospital Re-search Ethics Committee conditions.Funding for the evaluation will come from that portionof the CIHR corporate budget allocated to the systematicand regularly scheduled evaluation of the CIHR grantsand awards programs.DiscussionIn addition to the routine challenges of undertaking acomplex evaluation with multiple data sources and sta-keholders, there may be some unique ones related tothis evaluation. In particular, an evaluation that activelyengages program owners may raise concerns about itsindependence. Another potential challenge relates toconflicts that may arise during the evaluation betweenmembers of the Evaluation Working Group, and howthey should be resolved.To minimize the risk of any biases being introduced inthe evaluation, we built in a number of checks and bal-ances in our protocol design. However, it should benoted that CIHR’s current approach to evaluation doesinvolve engagement of the program owners (i.e., staffwho develop and administer funding opportunities) inreviewing data and recommendations. These program-owners may have vested interests in the program, andmay wish to influence the recommendations. CIHR’sgovernance structure is designed to minimize the occur-rence of this. An oversight committee exists that reviewsall evaluation plans and reports to ensure that the ap-propriate methodologies are used, analysis is undertaken,and that recommendations are supported by evaluationdata. This process is in place for the KT evaluation weare proposing to undertake.Another way we have designed the evaluation tominimize the introduction of bias is the expert reviewpanel. This panel will review the data and the interpreta-tions developed by the Evaluation Working Group andoffer an independent opinion on the analysis. Ultimatelythough, it is the integrity of the people involved thatmust be relied upon to ensure the evaluation is under-taken in a rigorous and transparent fashion. CIHR eva-luators and other staff operate under a government-widecode of conduct and perform their duties with profes-sionalism and due diligence, and the Evaluation WorkingGroup members have all be sensitized to reflect on theirindividual potential biases.While disagreements or tensions among the Evalu-ation Working Group are not anticipated given the goodworking relationships already existing between the mem-bers of the group, we do recognize that during thecourse of the evaluation, differences of opinion may ariseMcLean et al. Implementation Science 2012, 7:57 Page 13 of 16http://www.implementationscience.com/content/7/1/57given the different perspective of group members. Wehave agreed that ongoing open and frank discussionsduring our meetings will be a mechanism for addressingany conflicts. Should the Evaluation Working Group failto be able resolve conflicts on its own, these issues willbe brought to the oversight committee.In sum, the protocol and results of this evaluation willbe of interest to those engaged in the theory, practice, andevaluation of KT. The dissemination of the study protocoland results to both practitioners and theorists will help tofill a gap in knowledge in three areas: the role of a fundingagency in facilitating KT, the outcomes and impacts of KTfunding interventions, and how KT can best be evaluated.EndnotesaFor further detail on the TBS policy suite see: http://www.tbs-sct.gc.ca/cee/pol-eng.aspb DE and KTS funding opportunities are distinct pro-grams designed to support end of grant KT.c KT research projects are funded through CIHR’sOpen Operating Grant program.d An external expert on KT (MM) was involved ineach step of the evaluation planning process and sat as amember of the evaluation working group.e This excludes partnership programs. The Partner-ships Branch of CIHR was consulted as a part of theplanning phase of this evaluation. Results of this consult-ation indicated that the evaluation of partnerships pro-grams within the portfolio would not yield useful resultsfor program management in the context of a KT levelevaluation.f Commercialization programs are not included in theevaluation as they fit into a separate envelope of CIHRKT focus and strategy.g A logic model is an illustrative tool used to provide asimple, visual representation of the theory of change ofan intervention.h The measurement of a counterfactual group is a nat-ural and social science method of discerning causality. Inour case, a group of program participants will be com-pared to a group of individuals who did not participate inthe program. Comparison between the two groups againstappropriate evaluation questions will allow for the claimsabout the attribution that programs have had towardresponses to these questions. It should be noted that sam-pling techniques and the statistical significance of resultsmust be carefully considered in this method.Logic model narrativeKT has two bodies of officers, strategic leads and pro-gram officers. The activities of the KT suite of programsat CIHR have many similarities and therefore, aredescribed together, except for A5.Strategic leadsStrategic leads research, design and implement CIHR’sKT strategies. These activities include: Designing programs and funding opportunities,including the formulation and modification ofprogram regulations and processes Designing literature Offering training opportunities Conducting Research in the area of KT Running KT events Promotion and communication activitiesProgram officers A1-A4: Administration, including application processing,organization of peer or merit review, notificationand post-award notification MPD and KTR programs have peer reviewcompetition Knowledge Synthesis, PHSI and K2Aprograms have merit review competitions Monitoring of performance of programactivities and outcomesKTR review panel (A5) The KT research panel reviews all KT relatedapplications received in Open Operating GrantscompetitionsOutputs O1 - Knowledge synthesis grants awarded: Knowledge synthesis grants provide funding toresearchers who intend to produce scopingreviews or syntheses that meet the needs ofknowledge users in all areas of health. Grants are expected to increase the capacity ofresearchers to identify new, relevant avenues forexploration that have not yet been investigated thatrespond to decision makers/knowledge users’ needs. O2 - PHSI grants awarded: Researchers and decision makers enter intopartnerships to conduct applied and policy-relevanthealth systems and services research that respondto the needs of healthcare decision makers.McLean et al. Implementation Science 2012, 7:57 Page 14 of 16http://www.implementationscience.com/content/7/1/57 Partnerships can be project specific (partnersthat the researchers identify themselves withand with whom they negotiate) or competitionspecific (CIHR negotiated competitionpartnerships). Partners providing financialassistance are not required to be teammembers. O3 - K2A grants awarded: Grants are awarded to move knowledge intoaction by linking researchers and knowledgeusers located in the same community orregion. The development, implementation and evaluationof cutting-edge KT research and approaches arealso supported. O4 - DE and KTS grants awarded: Grants are awarded to support meetings, and/ordissemination activities consistent with themandate of CIHR and relevant CIHR Institutes,Initiatives or Branches. Dissemination Events support the organizationof events focused on the communication ofhealth research evidence. The KT Supplement supports KT activitiesthat follow a CIHR grant/award where furtherdissemination is appropriate. O5 - KT research grants awarded: Grants fund research directed toward developingtheory, evidence and innovation to define thedeterminants, implementation and uptake ofhealth research evidence into practice.Immediate outcomes IMM 01 - Meaningful knowledge user andresearcher partnerships established Partnerships between researchers and knowledgeusers are established, with knowledge users activein the research process IMM 02 – Knowledge generated Funded projects result in the generation ofknowledge. Quality knowledge includes syntheses of relatedresearch findings IMM 03 - Relevant research results are disseminatedand/or applied by partners and knowledge users Researchers and knowledge users work togetherto address relevant research questions and toexchange and apply knowledge to solve healthand health system problems. This results inresearch findings that are relevant to theknowledge user partners. IMM 04 - Advancement of KT science KT-funded grants advance the knowledge of KTin areas such as new approaches to KT,innovative KT tools, research into new strategiesfor facilitating the translation of findings intopractice, etc.Intermediate outcomes INT 01 - Knowledge users and researchers learnfrom each other: Researchers/knowledge users areactive in post-research knowledge translationactivities. Knowledge users are well informed byrelevant research. By bringing both researchers and knowledgeusers together, it is expected that research resultswill translate to actions that strengthen Canada'shealthcare system and/or improve the health ofCanadians KT capacity is developed, increasing the KTexpertise in Canada INT 02 - Knowledge users are informed by relevantresearch: Application of research findings byknowledge users The inclusion of knowledge users within theresearch process fosters greater ownership amongknowledge users. This results in improved ratesof application by knowledge users. Application includes the awareness of findingsamong knowledge users, influence/inclusion ofresearch findings in policy decisions, adoption offindings into practice, etc. INT 03 - Generalizable knowledge is created anddisseminated Research results from funded studies are madewidely applicable and disseminated outside thesphere of knowledge users and researchersdirectly related to the project.McLean et al. Implementation Science 2012, 7:57 Page 15 of 16http://www.implementationscience.com/content/7/1/57Long term outcomeUltimately, the KT suite of funding programs areintended to facilitate the translation of research into ap-plication in society at large, resulting in the improvedhealth for Canadians, more effective health services andproducts and a strengthened Canadian health-care sys-tem. The KT suite of funding programs are aimed towork together to improve KT capacity in Canada, im-prove knowledge of KT and integrate both researchersand knowledge users in the research process, improvingthe relevance and timeliness of research findings.AbbreviationsCIHR: Canadian Institutes of Health Research; KT: Knowledge Translation;TBS: Treasury Board Secretariat; PHSI: Partnerships for Health SystemsImprovement; K2A: Knowledge to Action; iKT: Integrated KnowledgeTranslation; DE: Dissemination Events; KTS: Knowledge TranslationSupplement; OOGP: Open Operating Grants Program; EIS: ElectronicInformation System.Competing interestsRKDM is an Evaluator at CIHR. As Principal Evaluator, RKDM is responsible forthe production of evaluation reporting for program development andmeeting requirements of TBS. IDG is the Vice President of KnowledgeTranslation and Public Outreach at CIHR. In this role and in the role ofPrincipal Evaluation User, IDG is ultimately responsible for the KT fundingprogramming being evaluated. KB is an Evaluator at CIHR. YC is SeniorKnowledge Translation Specialist at CIHR. SEC is Knowledge TranslationManager at CIHR’s Institute of Gender and Health. MM is chair of CIHR’s K2Amerit review panel. CM is a Junior Evaluator at CIHR. RM is Director of CIHR’sKT Branch. AM is Manager, KT Initiatives at CIHR. DP is Manager of CIHR’sEvaluation Unit. JMT is a Senior Advisor at CIHR. JT is a Junior Evaluator atCIHR.Authors' contributionsAll authors contributed to study conception and participated in criticallyappraising and revising the intellectual content of the manuscript. Allauthors read and approved the final manuscript. RKDM is Principal Evaluatorfor this study. IDG is Principal Evaluation User for this study.Author details1Evaluation Unit, Canadian Institutes of Health Research, 160 Elgin Street,Ottawa, ON, Canada. 2Knowledge Translation Portfolio, Canadian Institutes ofHealth Research, 160 Elgin Street, Ottawa, ON, Canada. 3School of Nursing,Faculty of Health Sciences, University of Ottawa, 451 Smyth Road, Ottawa,ON, Canada. 4Institute of Gender and Health, Canadian Institutes of HealthResearch, University of British Columbia, 6190 Agronomy Road, Vancouver,BC, Canada. 5School of Nursing, University of Northern British Columbia, 3333University Way, Prince George, BC, Canada.Received: 13 March 2012 Accepted: 5 June 2012Published: 22 June 2012References1. Agency for Health Research and Quality: Translating research into practice(TRIP)-II. Rockville, MD: AHRQ; Publication No. 01-P017, March 2001; [http://www.ahrq.gov/research/trip2fac.htm] Accessed Feb 11th, 2012.2. Mitton C, Adair CE, McKenzie E, Patten SB, Perry BW: Knowledge Transferand Exchange: Review and Synthesis of the Literature. The MilbankQuarterly 2007, 85(4):729–68.3. Straus SE, Tetroe J, Graham I: Defining knowledge translation. CanadianMedical Association 2009, 181(3–4):165–168.4. Tetroe J, Graham ID, Foy R, Robinson N, Eccles M, Wensing M, Durieux P,Legare F, Nielson CP, Adily A, Ward JE, Porter C, Shea B, Grimshaw JM:Health Research Funding Agencies’ Support and Promotion ofKnowledge Translation: An International Study. The Milbank Quarterly2008, 86(1):125–55.5. Canadian Institutes of Health Research: More about Knowledge Translationat CIHR. http://www.cihr-irsc.gc.ca/e/39033.html] Accessed Feb 11th, 2012.6. Government of Canada: Government of Canada Bill C-13. http://www2.parl.gc.ca/HousePublications/Publication.aspx?pub=bill&doc=C-13&parl=36&ses=2&File=28&Language=E] Accessed February 13, 2012.7. Lavis JN: Research, public policy, and knowledge-translation processes:Canadian efforts to build bridges. Journal of Continuing Education inHealth Professions 2006, 26(1):37–45.8. Canadian Institutes of Health Research: Funding Opportunity: KnowledgeSynthesis. http://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=1181&&view=currentOpps&org=CIHR&type=AND&resultCount=25&sort=program&all=1&masterList=true] Accessed February 11, 2012.9. Canadian Institutes of Health Research: Merit Review. http://www.cihr-irsc.gc.ca/e/39537.html] Accessed February 11, 2012.10. Canadian Institutes of Health Research: About PHSI. http://www.cihr-irsc.gc.ca/e/34348.html] Accessed February 11, 2012.11. Canadian Institutes of Health Research: Funding Opportunity: OperatingGrant: Knowledge to Action, Local Research-User Interaction. http://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=170&view=browseArchive&browseArc=true&progType=CIHR-12&type=AND&resultCount=25&all=1] Accessed February 11, 2012.12. Canadian Institutes of Health Research: Funding Opportunity: Meetings,Planning and Dissemination Grant. http://www.researchnet-recherchenet.ca/rnr16/vwOpprtntyDtls.do?prog=1191&view=search&incArc=true&org=CIHR&sponsor=CIHR-22&progType=CIHR-8&launchYear=2010&type=AND&resultCount=25]Accessed February 11, 2012.13. McDavid JC, Hawthorne LRL: Program Evaluation and PerformanceMeasurement: An Introduction to Practice. Thousand Oaks, CA: SagePublications; 2006.14. Patton MQ: Utilization-focused Evaluation. Thousand Oaks, CA: SagePublications; 2008.15. Carden F: Knowledge to Policy: Making the most of development research.New Delhi, India: Sage Publications; 2009.16. Gerring J: Is There a (Viable) Crucial Case Method? Comparative PoliticalStudies 2007, 40(3):231–253.doi:10.1186/1748-5908-7-57Cite this article as: McLean et al.: Understanding the performance andimpact of public knowledge translation funding interventions: Protocolfor an evaluation of Canadian Institutes of Health Research knowledgetranslation funding programs. Implementation Science 2012 7:57.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitMcLean et al. Implementation Science 2012, 7:57 Page 16 of 16http://www.implementationscience.com/content/7/1/57


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items