UBC Faculty Research and Publications

Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses Colquhoun, Heather L; Carroll, Kelly; Eva, Kevin W; Grimshaw, Jeremy M; Ivers, Noah; Michie, Susan; Sales, Anne; Brehaut, Jamie C Sep 29, 2017

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13012_2017_Article_646.pdf [ 416.17kB ]
JSON: 52383-1.0355883.json
JSON-LD: 52383-1.0355883-ld.json
RDF/XML (Pretty): 52383-1.0355883-rdf.xml
RDF/JSON: 52383-1.0355883-rdf.json
Turtle: 52383-1.0355883-turtle.txt
N-Triples: 52383-1.0355883-rdf-ntriples.txt
Original Record: 52383-1.0355883-source.json
Full Text

Full Text

RESEARCH Open AccessAdvancing the literature on designing auditand feedback interventions: identifyingtheory-informed hypothesesHeather L. Colquhoun1* , Kelly Carroll2, Kevin W. Eva3, Jeremy M. Grimshaw2,4, Noah Ivers5, Susan Michie6,Anne Sales7 and Jamie C. Brehaut2,8AbstractBackground: Audit and feedback (A&F) is a common strategy for helping health providers to implement evidenceinto practice. Despite being extensively studied, health care A&F interventions remain variably effective, with overalleffect sizes that have not improved since 2003. Contributing to this stagnation is the fact that most health care A&Finterventions have largely been designed without being informed by theoretical understanding from the behavioraland social sciences. To determine if the trend can be improved, the objective of this study was to develop a list oftestable, theory-informed hypotheses about how to design more effective A&F interventions.Methods: Using purposive sampling, semi-structured 60–90-min telephone interviews were conducted withexperts in theories related to A&F from a range of fields (e.g., cognitive, health and organizational psychology,medical decision-making, economics). Guided by detailed descriptions of A&F interventions from the health careliterature, interviewees described how they would approach the problem of designing improved A&F interventions.Specific, theory-informed hypotheses about the conditions for effective design and delivery of A&F interventionswere elicited from the interviews. The resulting hypotheses were assigned by three coders working independentlyinto themes, and categories of themes, in an iterative process.Results: We conducted 28 interviews and identified 313 theory-informed hypotheses, which were placed into 30themes. The 30 themes included hypotheses related to the following five categories: A&F recipient (seven themes),content of the A&F (ten themes), process of delivery of the A&F (six themes), behavior that was the focus of theA&F (three themes), and other (four themes).Conclusions: We have identified a set of testable, theory-informed hypotheses from a broad range of behavioraland social science that suggest conditions for more effective A&F interventions.This work demonstrates the breadth of perspectives about A&F from non-healthcare-specific disciplines in a waythat yields testable hypotheses for healthcare A&F interventions. These results will serve as the foundation forfurther work seeking to set research priorities among the A&F research community.Keywords: Audit and feedback, Implementation science, Knowledge translation, Theory* Correspondence: heather.colquhoun@utoronto.ca1Department of Occupational Science and Occupational Therapy, Universityof Toronto, 160-500 University Ave, Toronto, Ontario M5G 1V7, CanadaFull list of author information is available at the end of the article© The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Colquhoun et al. Implementation Science  (2017) 12:117 DOI 10.1186/s13012-017-0646-0BackgroundAudit and feedback (A&F), where data about specific as-pects of practice are summarized and fed back to practi-tioners to encourage practice change, is routinely andincreasingly employed in many clinical contexts. Themost recent Cochrane review on the effectiveness ofA&F interventions includes 140 trials [1] and shows thatsuch interventions yield modest (median adjusted riskdifference of 4.3% absolute increase) but variable (inter-quartile range of 0.5 to 16%) improvements in clinicalpractice. Despite so much collective experience, a cumu-lative analysis of estimates of effect by year indicatedthat effects’ sizes plateaued sometime around 2003 [2],suggesting our efforts to design effective A&F interven-tions are not improving.We propose that a key factor impeding progression to-wards more effective A&F healthcare interventions hasbeen a lack of theoretical understanding of the mecha-nisms underlying these interventions. We have shownthat theory is rarely invoked in the design of health careA&F interventions; less than 10% of A&F interventionsreport any theory guiding design of the intervention [3].Instead, the majority of current A&F interventions ap-pear to be guided by intuitive, non-theoretical ideasabout what might work [4]. Without the application oftheory, one cannot predict whether a successful inter-vention will generalize, learn much from failed interven-tions, or successfully tailor interventions to a newcontext [5, 6].Attempts to apply theory to A&F have focused on in-dividual theories from health and social psychology [7]and organizational science [8]. For example, a systematicreview using constructs informed by the Feedback Inter-vention Theory from organizational psychology [8] iden-tified specific constructs (e.g., frequency of feedback,patient-specific feedback) that are likely related to A&Feffectiveness. However, there is a much broader range oftheories that may suggest explicit, testable hypothesesabout how to optimize A&F interventions [9]. For ex-ample, using social interaction to increase what can belearned from A&F [10] could provide a rich source oftheory-informed concepts regarding how to design moreeffective A&F. Our team has argued that when seekingto identify theoretical mechanisms underlying complexinterventions, existing theories describing specific sub-components of these interventions may suggest ways toimprove the overall intervention [11]. This idea opensup a much wider range of theoretical perspectives thanhas currently been applied to healthcare A&F.We propose that the literature on healthcare A&F in-terventions will be advanced by consideration of ideasfrom a broad range of relevant theoretical traditions.Specific methods for incorporating theory from manydisciplines are limited, making it necessary to generatenovel approaches. Our approach involved identifyingand interviewing experts from specific, a priori definedtheoretical traditions, providing examples of healthcareA&F, and eliciting explicit, theory-informed hypothesesabout how that A&F could be improved. The objectiveof this study was to use this approach to develop a broadlist of testable, theory-informed hypotheses about howto improve A&F interventions.MethodsOur study used semi-structured, in-depth interviewswith theory experts to identify specific hypotheses and athematic analysis to organize the resulting hypothesesinto themes. Ethics approval was obtained from theOttawa Health Sciences Network Research Ethics Board.ParticipantsUsing purposive sampling, a preliminary list of theoryexperts from a priori defined relevant fields (i.e., cogni-tive psychology, social or health psychology, education,medical decision-making, industrial/organizationalpsychology, and economics) was developed by the studyteam. The list was developed based on the researchteam’s respective knowledge of theorists whose publica-tion history and impact made it clear they would providea useful perspective on how to improve A&F interven-tions. In order to qualify as a theory expert, the potentialparticipant had to have demonstrated expertise in one ormore relevant theories. The goal was to attempt cover-age across a broad range of fields deemed by the re-search team to have relevance to the study or use ofA&F. Interviewed participants were asked to suggestothers whose input they judged would be valuable basedon the interview experience (i.e., snowball sampling).We re-evaluated our list of potential participants on anumber of occasions to ensure that our sample included,to our knowledge, the most relevant theorists and abroad sample from a range of fields. Participants weregiven a $200 CAD honorarium for their participation.Describing our sample and related theoretical expertiseWe categorized each participant into a field based ontheir self-described primary area of expertise or discip-line. For the purposes of describing field coverage, ex-perts with extensive expertise in two of the a prioridefined fields were coded twice. We also created a sum-mary of the theories and/or theoretical concepts de-scribed by the participants as informing their workthrough analysis of (1) their answer to the first interviewquestion specific to their area of theoretical expertiseand (2) additional theories and/or theoretical conceptsdiscussed during the course of the interview.Colquhoun et al. Implementation Science  (2017) 12:117 Page 2 of 10The interview guide and interviewExperts were sent materials prior to the interview ses-sion, including the interview questions, a summary de-scription of four published A&F randomized trials, andthe relevant trial publications [12–15]. The four targetexamples were intended to represent a range of commonhealthcare A&F interventions and differed in many ways(i.e., whether the A&F was group or individual, whetherthe A&F was given to the target for behavior change,what the A&F was about, the use of target goals orbenchmarks, key educational messages, and the fre-quency of the A&F). Additional file 1 includes a descrip-tion of the four A&F interventions. Interviewees wereasked to read the material in preparation for theinterview.The interview guide (see Additional file 1) was pilottested in four interviews to establish the appropriateness,flow, and robustness of the guide. All interviews wereconducted by three members of the team (HLC, JCB,KC). Interviews were conducted by telephone, audio-recorded, and lasted between 60 and 90 min. Interviewscovered 2–4 of the A&F examples, depending on howthe conversations went. We planned to cease samplingonce we achieved coverage of our a priori defined fieldsand saturation for theme development (i.e., new inter-views generally fit within the current thematic codingframe with no new themes identified).The interview consisted of three main tasks. The firstasked the expert to describe their theoretical expertiseand the theories that guide their work. This allowed theparticipant to review their own foci prior to exploringthe A&F interventions and oriented the interviewers tothe participant’s specific theoretical expertise, jargon,and approach to the concept of A&F. The second taskinvolved one interviewer (HLC) explaining, one by one,and in detail, up to four A&F interventions. For each,the participant was asked to provide their initial open-ended reactions to each A&F intervention. They werethen asked to comment on aspects they liked or dislikedabout each intervention and how they would go aboutimproving it. In doing so, they were encouraged to de-scribe their input in terms of specific theory-based andspecified hypotheses for more effective A&F as much aspossible (as opposed to intuitive ideas on designing bet-ter A&F). Interviewers asked clarifying questions whenneeded and engaged in discussion aimed at understand-ing the hypotheses proposed and the theory behindthem. Interviewers sought to identify what theoreticalperspective led to each hypothesis. When the discussiondid not make it clear, this link was further sought as partof the member checking process. The third and finaltask was to ask the participant if they had any additionalthoughts on how best to design A&F interventions thatwere not discussed during the review of the examples.This task was intended to facilitate discussion of abroader range of hypotheses related to A&F effectivenessbeyond those invoked by the four specific examples.Member checking the hypothesesFollowing each interview, and using the audio-recordingand notes as a guide, one member of the research team(KC) developed a draft member checking document out-lining the testable hypotheses described in the interview.This document was used to confirm that the researchteam correctly understood the expert’s perspective. Atable was developed that summarized the concept oridea behind the hypothesis, the specific hypothesis, andwhere possible, relevant mechanisms of action, media-tors, outcomes, contextual factors and theories guidingthe hypothesis. The draft was iteratively reviewed andmodified by two other members of the research team(HLC, JCB) until all three agreed as to accuracy andcompleteness. The theory expert was then asked to re-view and use track changes to modify and add additionaldetail or clarity to the final document.Theme generationIn order to organize the hypotheses from all final inter-view documents, we used a process similar to the con-stant comparative method of data analysis (open coding)used in qualitative research [16]. Hypotheses were inde-pendently assigned to themes in an iterative process by3 coders (JCB, HLC, KC). This was done in blocks of 50or 100 randomly chosen hypotheses. A consensus meet-ing was held to review proposed themes and develop aninitial coding frame. After each block, we repeated theprocess and modified the coding frame with new themesas needed. Our focus was to get a clear understanding ofthe full range of hypotheses, not to keep the themes to aminimum. Prior to finalizing the themes, we identifiedand removed the hypotheses that were identical or du-plicated, resulting in a total number of unique hypoth-eses. If hypotheses were similar, but seemedconceptually different for any reason, we did not desig-nate the hypotheses as duplicate. This process was con-ducted by two team members separately (HLC, KC),followed by a consensus discussion for any discrepan-cies. The final thematic structure was confirmed by afourth member of the team (KWE). The final task in-volved grouping the 30 themes into general categories.This was conducted by three members of the team sep-arately (HLC, KC, KWE) followed by a consensus dis-cussion and confirmation by a fourth member of theteam (JCB).ResultsWe approached 47 theorists over a 1-year period.Twenty-eight (60%) agreed to participate and underwentColquhoun et al. Implementation Science  (2017) 12:117 Page 3 of 10a full interview. Five refused to participate; three weretoo busy, and two expressed lack of expertise. An add-itional 14 did not respond. Table 1 describes the disci-plines or fields of the participants. Eighteen of the 28participants were from the USA, five were from Canada,and the remaining five were from various countries inEurope. The discipline or field with the most partici-pants was cognitive psychology (n = 9), and the leastrepresented was human factors (n = 2).Table 2 describes the range of expertise described bythe participants as informing their work. Several partici-pants cited expertise in Goal Setting Theory [17, 18],Control Theory [19], Self-Regulation Theory [20], Self-Efficacy [21], and various behavior change and learningtheories. Together, there were over 100 different areas ofexpertise provided by the participants.Hypotheses generatedThe 28 interviews yielded a total of 389 hypotheses.After duplicates were removed, 313 unique hypothesesremained. These hypotheses were organized into a cod-ing framework with 30 themes (Table 3) across five cat-egories. The complete list of all 313 unique hypotheses,organized according to theme and category, can befound in Additional file 2.Related to the recipient (n = 63 hypotheses in seventhemes)The hypotheses and themes in this category pertain tothe reaction or perspective of the recipient of the A&F.The largest theme in this category was trustworthiness/credibility, which contained 14 hypotheses all outliningthe importance of considering the degree to which a re-cipient trusts the source of and/or data in the A&F (e.g.,A&F will be more effective if it is perceived to be with-out conflict of interest, when recommendations relatedto the A&F are based on good quality evidence). Thir-teen hypotheses related to motivation/intention issues ofthe recipient, such as using positive reinforcement (e.g.,A&F will be more effective over time if it isaccompanied with positive reinforcement to those whohave improved their performance). The theme recipientcharacteristics contained nine hypotheses, all related tohow attributes of the recipient of the A&F should informthe A&F design (e.g., A&F will be more effective forthose with a mastery goal orientation if it involves com-parison to others). Nine hypotheses outlined the import-ance of ensuring an understanding of recipient priorities(e.g., A&F will be more effective when targeted at behav-iors that the recipient feels are important to their profes-sional roles/responsibilities), and seven hypotheses werein the theme attack on self-identity and described howA&F needs to ensure that defensive reactions do nottake place (e.g., A&F will be more effective when mea-sures are used to prevent a defensive response—provid-ing other “reassuring” messages as well). The last twothemes (six and five hypotheses respectively) containedhypotheses related to how best to attract and maintainattention of the recipient (e.g., A&F will be more effect-ive if they engage the target’s attention), and how to de-sign A&F to maintain self-efficacy/control (e.g., A&F willbe more effective if the behavior is under the control ofthe recipient).Related to the behavior (n = 22 hypotheses in threethemes)All of the hypotheses and themes in this category werefocused on the behavior that the A&F intervention wasmeant to change. The largest theme was remove barrierswhich included 11 hypotheses that encouraged an un-derstanding of the specific barriers to the behavior (e.g.,A&F will be more effective if based on a barriers ana-lysis). There were seven hypotheses in the theme aboutaspects of the behavior that outlined conditions relatedto the behavior itself (e.g., A&F will be more effective forbehaviors that are easy compared to those that areharder to do). The last theme, decision processes or con-ceptual model, pertained to ensuring a good understand-ing of behavioral decision-making (e.g., A&F will bemore effective if designed with a clear understanding ofthe decision making process underlying the behavior tobe changed).Related to the content of the A&F (n = 145 hypotheses inten themes)All of the hypotheses and themes in this category werefocused on the content included in the A&F. The themewith the most hypotheses in this category was cognitiveload, which contained 33 hypotheses all related to redu-cing the amount of mental effort required to mentallyprocess the A&F. It included hypotheses related to over-all simplicity (e.g., A&F will be more effective if as fewgraphs as possible are presented), the display of the A&F(e.g., A&F will be more effective when color changes areTable 1 Expertise of participants by discipline or field and total,n = 28Discipline or field TotalCognitive psychology 9Education 8Medical decision-making 7Industrial organization or management 6Social or health psychology 5Medical education 5Economics 3Human factors 217 participants were categorized into two disciplines or fieldsColquhoun et al. Implementation Science  (2017) 12:117 Page 4 of 10Table 2 Range of self-described expertise and other areas of expertise by participantExpert Self-described expertise Other concepts/areas of expertise referred to during interview1. Brunswikian psychology, diagnostic judgments of physicians, useof vignettes containing clinical cuesLens modeling, evidence-based medicine, behavior change theory,face validity2. Behavior decision theory, methodological theory of informationintegration—what cues people pay attention toDiffusion of responsibility, norm theory, SMART goals, loss aversiontheory, scale compatibility, habituation theory, spacing effects3. Human factors, health communication and decision making, gestaltprinciples, information scienceTheories of attention, international design standards, prospect theory,loss aversion, cognitive load, constructivist learning theory,intrinsic/extrinsic motivation4. Self-assessment, behavior change, comparison models Guided reflection, learner centered agenda, teacher directed agenda5. Diagnostic reasoning of physicians, dual process models, informationdistortion (gestalt)Learning theories, theory of planned behavior, extrinsic motivation6. Human factors engineering, iterative design Theory of planned behavior, theories of operant conditioning, law ofeffect7. Cognitive psychology, judgment decision making framework,information processing, linguisticsTufte theory, incentives, Lake Woebegone effect in social psychology8. Applied work in medical decision making, hindsight bias Fast and frugal heuristics9. Cognitive psychology—how people reason, formulate judgments,and make decisions10. Personality, social, and health psychology, principles of feedbackcontrolSelf-regulation of behavior11. Cognitive psychology, learning, memory, lab research on feedback12. Behavioral economics, psychology, rational choice Individual limitations, motivations, ego, mastery, social comparison13. Organizational psychology, feedback research, feedback seekingbehavior, feedback environment framework, information processing,achievement goal theory, personalitySelf-motives framework, self-enhancement theory, self-determinationtheory, motivation, dual process models, serial position curve/memory14. Cognitive psychology, measurement, assessment, formative feedback,constructivism, active learning theoriesCognitive load, growth mindset work (Dweck), goal setting theory,display of quantitative information (Tufte), graph design15. Social psychology, attribution/dissonance theory, prospect theory,conflict and dispute resolution, study of influenceLewinian channel factor identification, self-perception theory, socialnorms theory, nudge theory (Thaler and Sunstein), motivation,Prospect Theory16. Methodology, resource management principal17. Bjork’s desirable difficulties18. Psychology, dual processes, affect and emotion, numeracy and aging19. Goal setting theory Cognitive load20. Social psychology, health communication strategies or healthdecision making and health behavior change, social cognitivetheory, theory of planned behavior, adoption process model,social comparison theory, classic theories of attitude and behaviorStudy of influence21. Control theory, self-regulation theory, goal setting theory, self-efficacy Reinforcement theory, partial reinforcement theory22. Industrial organizational psychology, work motivation, teamperformance, feedback from the standpoint of individual behaviorSelf-regulation, control theory, goals and actions, subjective expectedutility theory (theory of reasoned action, expectancy theory)23. Educational theory, learning theories, constructivism, socio-culturallearning theoriesReflective learning, motivation, peer learning, communities ofpractice, social learning, peer scaffolding, role modeling24. Applying psychological principals in clinical practice, sociologicallearning theory, sociocultural theory, feedback interventionsDiscourse theory, activity theory, complexity theory, achievementmotivation theory25. Psychology, economics, ethics, patient physician communication,treatment decision making26. Education research, feedback in education, social cultural theory27. Industrial organizational psychology, Power’s control theory(self-regulation theory), Carver and Scheier’s social cognitive theoryGain theory, implementation intentions28. Education, constructivist approach, basic notions of socialpsychology, multisource feedback or feedback to students fromsupervisors (Ross and Nesbett), social cultural theory (Vigotsky),humanist theory (Carl Rogers)Theories of behavior change, self-regulation, feedback interventiontheory (Kluger and Denisi), motivation theories, informed self-assessmentColquhoun et al. Implementation Science  (2017) 12:117 Page 5 of 10Table 3 Summary of hypotheses by theme and with examplesThemes(N = 30)# of hypotheses(N = 313)Example hypothesesA&F/A&F interventions will be more effective…Related to the recipient1. Trust/credibility 14 If it is perceived to be without conflict of interest; when recommendationsrelated to the A&F are based on good quality evidence2. Motivation/intention 13 If it is accompanied with positive reinforcement to those who have improvedtheir performance; when accompanied by incentive3. Recipient characteristics 9 For those with a mastery goal orientation if it involves comparison to others4. Recipient priorities 9 When targeted at behaviors that the target feels is important to their professionalroles/responsibilities5. Attack on self-identity 7 When measures are used to prevent a defensive response(providing other “reassuring” messages as well)6. Attract/maintain attention 6 If they engage the target’s attention7. Self-efficacy/control 5 If the behavior is under the control of the recipientRelated to the behavior8. Remove barriers 11 If they address barriers to change in behavior9. About aspects of behavior 7 For behaviors that are easy compared to those that are harder to do10. Decision processes orconceptual model4 If designed with a clear understanding of the decision making processunderlying the behavior to be changedRelated to the content of the A&F11. Cognitive load 33 If as few graphs as possible are presented; without unnecessary depth elements;if the graphical representations are clearly and consistently labeled; when colorchanges are purposeful and convey meaning; when presenting absolute numbers asopposed to percentages; when graphical clutter is removed; when focused on a few,most important behaviors12. Comparisons 26 When the benchmark comparison is justified to be a reasonable standard; when acomparator is provided; when multiple individual practice data is presented along withthe recipient’s data; if it involves a comparison to the self; if the comparator is specific tothe recipient’s own context/practice.13. Action plans/copingstrategies19 If clear direction on how to change behavior is provided14. Feedback specificity 16 If individual level provider data is provided; if patient-specific information is provided;if it is as specific as possible15. Goal setting 16 If it is accompanied by a goal that is specific16. Justify need for behavior change 10 If accompanied by evidence supporting the behavior change17. Cognitive influences 7 If emphasis is on what needs to be achieved (loss framing) as opposed to what wasachieved (gain framing).18. Nature of the data 6 If graphical representation displays the variability of data in order to indicate theerror or uncertainty19. Guide reflection 6 If it involves a personal reflection component20. Improving memory 6 If the reminder messages are presented in real time/point of care; if incorporatesan emotional message underlining the desired behaviorRelated to the delivery of the A&F21. A&F timing 20 If individual change data over time is provided; when presented multiple times;when presented at the time of decision making22. Social engagement 17 If they involve engaging recipients in social discussion about the A&F23. Knowledge/learning 13 If it creates opportunities to learn24. User-guided experience 6 When complex information is scaffolded to allow a recipient to get more informationif and when they want25. In-person A&F 2 When provided with human contact26. Responding to A&Fproviders2 If they allow the recipient an opportunity to indicate why a recommendedaction was not taken.Colquhoun et al. Implementation Science  (2017) 12:117 Page 6 of 10purposeful and convey meaning), and the content of theA&F (e.g., A&F will be more effective when focused onthe few, most important behaviors). Twenty-six hypoth-eses focused on comparisons including the use of bench-marks as comparisons in the A&F (e.g., A&F will bemore effective when the benchmark comparison is justi-fied to be a reasonable standard), comparisons in gen-eral, social comparisons (e.g., A&F will be more effectivewhen multiple individual physician practice data is pre-sented along with the recipients’ data), comparisons tothe self, and the specificity of the comparison. Nineteenhypotheses related to enabling action plans/coping strat-egies (e.g., A&F will be more effective if clear direction isprovided on how to change behavior). A&F specificity in-cluded hypotheses related to A&F being specific to theindividual, being patient specific, or around the ideallevel of specificity (e.g., A&F will be more effective if it isas specific as possible). The positive effect of goal settingwithin A&F was addressed in 16 hypotheses (e.g., A&F iswill be more effective if accompanied by a goal that isspecific). The remaining themes included hypotheses re-lated to ensuring that the A&F justifies the need for be-havior change, other cognitive influences, the nature ofthe data presented, designing the content such that itguides the recipient (guide reflection), and improvingmemory by using reminders.Related to the delivery of the A&F (n = 60 hypotheses insix themes)All of the hypotheses and themes in this category werefocused on the processes used when delivering the A&F,regardless of the content of the A&F. The largest themein this category included 20 hypotheses related to A&Ftiming. This included hypotheses about providing A&Fover time (e.g., A&F will be more effective if individualchange data over time is provided), multiple times, andother timing-related issues (e.g., A&F will be more ef-fective when presented at the time of decision making).The theme of social engagement had 17 hypotheses re-lated to engaging recipients in social discussion aboutthe A&F (e.g., A&F interventions will be more effectivewhen they incorporate facilitated social discussionsabout the A&F). Thirteen hypotheses focused onknowledge/learning (e.g., A&F that creates opportunitiesto learn will be more effective). The remaining themesincluded hypotheses related to allowing the recipient tocontrol how they access the A&F (user-guided experi-ence), delivering the A&F in person (in-person feedback),and delivering the A&F such that the recipient is askedto respond to the A&F (responding to feedbackproviders).Other (n = 23 hypotheses in four themes)This category includes three themes that did not relateto the four categories above, as well as a grouping of tensingle hypotheses that did not relate to any theme. Therewere seven hypotheses that outlined the importance ofconsidering the opportunity costs of the A&F (e.g., A&Fis more effective when there are few costs to change be-havior), four hypotheses that related to the environment(e.g., A&F is more effective if the environment encour-ages the desired behavior as the default), and two hy-potheses related to development process involvement, orincluding the recipients in the design of the A&F (e.g.,A&F is more effective when recipients have been in-volved in the design of the A&F). A notable single hy-pothesis was that A&F will be more effective if madepublicly available.DiscussionIn an effort to broaden the range of theoretical perspec-tives to apply to health care A&F, we successfully inter-viewed 28 theory experts from a broad set of theoreticalperspectives and fields and created a list of testable,theory-informed hypotheses about how healthcare A&Finterventions might be improved. We developed a list of313 unique hypotheses in 30 themes. To our knowledge,this is the first explicit effort to bring theory from manydifferent relevant disciplines to the problem of optimiz-ing health care A&F interventions. Our approach wassuccessful in yielding new hypotheses that are not cur-rently captured in existing A&F theories and that, to ourknowledge, have not been tested in evaluation studies ofA&F [1]. The hypotheses and/or themes presented inthis paper will form the basis of a future prioritizationexercise designed to support a coherent, theory-guidedTable 3 Summary of hypotheses by theme and with examples (Continued)Other27. Opportunity costs 7 When there are few costs to change behavior28. Environment 4 If the environment encourages the desired behavior as the default.29. Development processinvolvement2 When recipients have been involved in the design of the A&F30. Single hypotheses 10 If they imply some kind of extended commitment; if the recipient generates aresponse immediately prior to receiving the A&F; if the goal is made public; if itis provided to the intended target for behavior change; it includes multiple modes ofinformation (e.g., pictures and text)Colquhoun et al. Implementation Science  (2017) 12:117 Page 7 of 10research agenda for optimizing A&F as an implementa-tion intervention. Developing this agenda will be thenext step of this work.Currently, both intervention development and evalu-ation of A&F interventions are driven primarily by theintuitions of individual investigators [3]. This work pro-vides an initial step towards a more theory-guided sci-ence of A&F development and evaluation. This sort oforganized approach to evaluation has been highlightedas an essential future research direction for this field [2].We expect this work to not only help prioritize researchdirections for the field but also encourage ambitiouslarge-scale trials comparing multiple approaches to A&F[22] and to assist A&F laboratories tasked with exploringand designing innovative interventions involvingA&F [23].The number of potential hypotheses identified and therange of theories and theoretical concepts discussed un-derscores the complexity and number of potential mech-anisms underlying effective A&F. We found thatconstructs from well-known theories specific to A&Fwere well represented in our hypotheses. For example,constructs from the Feedback Intervention Theory [8]such as feedback timing (i.e., the more frequent the bet-ter), the importance of goal setting, and the role of per-sonality on the reaction to feedback were clearlyrepresented in our list. Constructs from Hysong’s Modelof Actionable Feedback [24] were also represented (e.g.,feedback needs to be timely, individualized, non-punitive, and customizable). Importantly, however, manyof the hypotheses generated (e.g., related to social en-gagement, trustworthiness/credibility, removing barriers,justifying need for behavior change, nature of the data,environment) are not represented in A&F-specific theor-ies and instead stem from theories that might be seen asoverlapping with A&F or target components of A&F, ra-ther than describing A&F itself. Consider the following:A&F will be more effective if noun descriptors ratherthan verbs are used in messaging—‘do not be an overprescriber’ versus ‘please prescribe less’.A&F will be more effective if it incorporates a gamingapproach.A&F will be more effective if information aboutopportunity costs is included; A&F will be moreeffective when recipients have been involved in thedesign of the A&F.A&F interventions will be more effective if they involveengaging recipients in social discussion about the A&F.None of these hypotheses are specified as part of exist-ing theories of A&F, but nevertheless suggest potentiallyinnovative ways to improve this class of interventions.These findings suggest that there may be more to belearned about the A&F process if we allow ourselves toincorporate constructs and mechanisms from othertheories [11].The initial focus of this work was to generate testablehypotheses that were clearly and closely tied to the spe-cific theories in which our participants were most ex-pert. Despite our efforts in both the interviews and inthe member checking process, participants often hadconsiderable difficulty and/or showed reluctance to ex-plicitly tie hypotheses to specific theories. The theory ex-perts were probed about their specific theoreticalorientation and were told to focus on hypotheses with atheoretical basis (as opposed to intuitive ideas), yet it be-came clear that the hypotheses being generated variedsubstantially in terms of how clearly they could bemapped onto a specific theory. For example, hypothesesrelated to goal setting could easily be mapped to theory[17]; in contrast, many hypotheses related to cognitiveload (e.g., remove graphical clutter, label consistentlyand clearly) were less likely to be ascribed to a specifictheory by our experts. In general, while we believe mostexperts adhered to our instructions and tried to generatetheory-based hypotheses, it is possible that a subset ofour hypotheses are better described as the intuitionsbased on the experience of theory experts [25], ratherthan hypotheses clearly predicted by theory. Regard-less, we see this work as being a possible and poten-tially more direct route towards guiding the A&Fresearch community towards better interventions thanthe current serendipity-driven and intuition-basedapproach [4].With such a vast list, we felt obliged to organize intothemes and categories, yet without a definitive taxonomyof A&F, we proposed only a simple, descriptive structurebased on team consensus. Our efforts were designed toorganize the themes into a manageable number of cat-egories and not to propose a framework with any im-plied structure of importance or relevance outside of thesummary. Our category scheme (relevant to the recipi-ent, the behavior, the content, and the delivery of theA&F) is one way to frame important A&F elements;others have organized them differently [2, 4, 24, 26, 27].A definitive taxonomy of A&F interventions would helpstandardize how A&F interventions are designed, de-scribed, and reported.A number of additional challenges of this work war-rant consideration. First, these interviews were extremelylabor intensive and challenging, often involving unfamil-iar jargon; it is therefore likely that some of the subtle-ties of the various concepts discussed were lost, despitean extensive and iterative member checking process. Wefeel that part of the innovation of this approach was thefocus of the interviews on testable hypotheses, which fa-cilitated in-depth discussion between interviewers andColquhoun et al. Implementation Science  (2017) 12:117 Page 8 of 10interviewees despite quite different expertise. Anotherpossible limitation stems from the 4 specific A&F inter-ventions that were the focus of discussions; different hy-potheses may have been generated had we chosendifferent A&F examples. Indeed, most of our discussionsstarted out with the participant talking about display is-sues specific to the individual example. We sought toovercome this potential demand characteristic by specif-ically asking for thoughts on A&F in general as opposedto the examples, but it is very possible that the examplesdirected interviewee attention towards specific issues.This is one of the reasons why we believe it would be anerror to equate frequency with which a hypothesis wasmentioned with its potential importance or priority forstudy. Again, however, we see this design choice as partof the innovation of our approach, as these examples fa-cilitated in depth discussion between people of very dif-ferent expertise. Finally, we cannot guarantee that oursample covered all relevant disciplines, theoretical per-spectives, and geographical areas (i.e., the sample wasexclusively North American and European). This is,however, the largest compilation of A&F-relevant hy-potheses to date. The approach provided an extensivelist of testable hypotheses that would have been far moredifficult (or impossible) to achieve through other ap-proaches (e.g., literature review), and that includes hy-potheses novel to the healthcare A&F literature.ConclusionThe development of the scientific basis of A&F inhealthcare appears to have stagnated; we are not devel-oping more effective A&F interventions than we were20 years ago. We developed a methodology that wouldallow this area to be informed by a much wider range oftheoretical work than was possible previously. Our list oftheory-informed hypotheses will be an important foun-dation for moving this literature forward, enablingprioritization exercises, head-to-head trials where thearms are informed by theory and not just investigator in-tuition, more comprehensive theoretical descriptions ofA&F processes, and ultimately more consistently effect-ive A&F interventions. With such a list, the field will bebetter positioned to systematically guide the continuedevolution of this important intervention.Additional filesAdditional file 1: The interview guide (or supplemental file with theappendix below in the paper). (DOCX 369 kb)Additional file 2: All 313 hypotheses organized in 30 themes.(DOCX 37 kb)AbbreviationA&F: Audit and feedbackAcknowledgementsDr. Grimshaw holds a Canada Research Chair in Health Knowledge Transferand Uptake.FundingThis study was funded by the Canadian Institutes of Health Research(#130354).Availability of data and materialsThe datasets generated during and/or analyzed during the current study areavailable from the corresponding author on reasonable request.Authors’ contributionsHLC and JCB contributed to the conception and design of the study, theacquisition, analysis and interpretation of data, and drafted the manuscript.KC contributed to the acquisition, analysis and interpretation of data, anddrafted the manuscript. KWE, JMG, NI, SM, and AS contributed to theinterpretation of data in this study. All authors contributed edits to, read, andapproved the final version of the manuscript.Ethics approval and consent to participateEthics approval was obtained from the Ottawa Health Sciences NetworkResearch Ethics Board.Consent for publicationNot applicable.Competing interestsThe authors declare that they have no competing interests.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1Department of Occupational Science and Occupational Therapy, Universityof Toronto, 160-500 University Ave, Toronto, Ontario M5G 1V7, Canada.2Ottawa Hospital Research Institute, Clinical Epidemiology Program, TheOttawa Hospital, General Campus, Centre for Practice Changing Research,501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada. 3Centre for HealthEducation Scholarship, Department of Medicine, University of BritishColumbia, Vancouver, BC V5Z 4E3, Canada. 4Department of Medicine,University of Ottawa, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada.5Department of Family Medicine, Women’s College Hospital, Toronto, ONM5S 1B2, Canada. 6Department of Clinical, Educational and HealthPsychology, London, University College London, London WC1E 6BT, UK.7Department of Learning Health Sciences, University of Michigan MedicalSchool, VA Ann Arbor Healthcare System, Health Services Research andDevelopment, Ann Arbor, MI 48109, USA. 8School of Epidemiology, PublicHealth and Preventive Medicine, University of Ottawa, 451 Smyth Road,Ottawa, Ontario K1H 8M5, Canada.Received: 21 June 2017 Accepted: 7 September 2017References1. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD,O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback:effects on professional practice and healthcare outcomes. CochraneDatabase Syst Rev. 2012;6(6):Art. No.: CD000259. doi:10.1002/14651858.CD000259.pub3.2. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O'Brien MA, French SD,Young J, Odgaard-Jensen J. Growing literature, stagnant science? Systematicreview, meta-regression and cumulative analysis of audit and feedbackinterventions in health care. J Gen Intern Med. 2014;29(11):1534–41.3. Colquhoun HL, Brehaut JC, Sales A, Ivers N, Grimshaw J, Michie S, Carroll K,Chalifoux M, Eva KW. A systematic review of the use of theory inrandomized controlled trials of audit and feedback. Implement Sci.2013;8:66.Colquhoun et al. Implementation Science  (2017) 12:117 Page 9 of 104. Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, ChalifouxM, Eva K, Brehaut J. Reporting and design elements of audit and feedbackinterventions: a secondary review. BMJ quality & safety. 2016;5. Eccles M, Grimshaw J, Walker A, Johnston M, Pitts N. Response to "The OFFtheory of research utilization". J Clin Epidemiol. 2005;58:117–8.6. Michie S, Johnston M. Theories and techniques of behaviour change:Developing a cumulative science of behaviour change. Health Psychol RevTaylor & Francis Group. 2012;6(1):1–6.7. Gardner B, Whittington C, McAteer J, Eccles MP, Michie S. Using theory tosynthesise evidence from behaviour change interventions: the example ofaudit and feedback. Soc Sci Med. 2010;70(10):1618–25.8. Kluger AN, DeNisi A. The effects of feedback interventions on performance:a historical review, a meta-analysis, and a preliminary feedback interventiontheory. Psychol Bull. 1996;119:254–84.9. Shute VJ. Focus on formative feedback. Rev Educ Res. 2008;78(1):153–89.10. Boud D, Molloy E. Rethinking models of feedback for learning: thechallenge of design. Assessment & Evaluation in Higher Education. 2013;38(6):698–712.11. Brehaut JC, Eva KW. Building theories of knowledge translationinterventions: use the entire menu of constructs. Implement Sci. 2012;7:114.12. Thomas RE, Croal BL, Ramsay C, Eccles M, Grimshaw J. Effect of enhancedfeedback and brief educational reminder messages on laboratory testrequesting in primary care: a cluster randomised trial. Lancet. 2006;367(9527):1990–6.13. Wadland WC, Holtrop JS, Weismantel D, Pathak PK, Fadel H, Powell J.Practice-based referrals to a tobacco cessation quit line: assessing theimpact of comparative feedback vs general reminders. Ann Fam Med. 2007;5(2):135–42.14. Beck CA, Richard H, Tu JV, Pilote L. Administrative data feedback foreffective cardiac treatment: AFFECT, a cluster randomized trial. J Am MedAssoc. 2005;294(3):309–17.15. Tierney WM, Hui SL, McDonald CJ. Delayed feedback of physicianperformance versus immediate reminders to perform preventive care.Effects on physician compliance. Med Care. 1986;24(8):659–66.16. Creswell J, Plano Clark V. Designing and conducting mixed methodsresearch. California: Sage Publications; 2007.17. Locke EA, Latham GP. A theory of goal setting & task performance.Englewood Cliffs: Prentice Hall; 1990.18. Locke EA, Latham GP. Building a practically useful theory of goal settingand task motivation. A 35-year odyssey. Am Psychol. 2002;57(9):705–17.19. Carver C, Scheier MF. Attention and self regulation: a control theory tohuman behavior. New York: Springer-Verlag; 1981.20. Carver C, Scheier MF. On the self-regulation of behaviour. New York:Cambridge University Press; 1998.21. Bandura A. Self-efficacy: the exercise of control. New York: Freeman; 1997.22. Elouafkaoui P, Young L, Newlands R, Duncan EM, Elders A, Clarkson JE,Ramsay CR, Translation Research in a Dental Setting Research MethodologyG. An audit and feedback intervention for reducing antibiotic prescribing ingeneral dental practice: the RAPiD cluster randomised controlled trial. PLoSMed. 2016;13(8):e1002115.23. Ivers NM, Grimshaw JM. Reducing research waste with implementationlaboratories. Lancet. 2016;388(10044):547–8.24. Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practiceguideline adherence: making feedback actionable. Implement Sci. 2006;1:9.25. Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and itsuse in improvement. BMJ quality & safety. 2015;24(3):228–38.26. Ilgen DR, Fisher CD, Talyor SM. Consequences of individual feedback onbehavior in organizations. J Appl Psychol. 1979;64(4):349–71.27. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, Ivers N,Grimshaw JM. Practice feedback interventions: 15 suggestions foroptimizing effectiveness. Ann Intern Med. 2016;164(6):435–41. •  We accept pre-submission inquiries •  Our selector tool helps you to find the most relevant journal•  We provide round the clock customer support •  Convenient online submission•  Thorough peer review•  Inclusion in PubMed and all major indexing services •  Maximum visibility for your researchSubmit your manuscript atwww.biomedcentral.com/submitSubmit your next manuscript to BioMed Central and we will help you at every step:Colquhoun et al. Implementation Science  (2017) 12:117 Page 10 of 10


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items