UBC Faculty Research and Publications

Managing the incidence of selective reporting bias: a survey of Cochrane review groups Reid, Emma K; Tejani, Aaron M; Huan, Lawrence N; Egan, Gregory; O’Sullivan, Cait; Mayhew, Alain D; Kabir, Monisha Jun 13, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13643_2015_Article_70.pdf [ 1.07MB ]
JSON: 52383-1.0223373.json
JSON-LD: 52383-1.0223373-ld.json
RDF/XML (Pretty): 52383-1.0223373-rdf.xml
RDF/JSON: 52383-1.0223373-rdf.json
Turtle: 52383-1.0223373-turtle.txt
N-Triples: 52383-1.0223373-rdf-ntriples.txt
Original Record: 52383-1.0223373-source.json
Full Text

Full Text

RESEARCH Open AccessManaging the incidence ofgotenot fully presented based on the nature or direction ofthe results [1]. SRB may include the incomplete publica-demonstrated that 40–62 % of trials had changed, omit-ted or introduced one or more primary outcomes forg twoignifi-comesReid et al. Systematic Reviews  (2015) 4:85 DOI 10.1186/s13643-015-0070-yAvenue, Vancouver, BC V5Z 1M9, CanadaFull list of author information is available at the end of the articlegives rise to a potential over- or underestimation oftreatment effects or harms [2–4, 7].* Correspondence: emma.reid@vch.ca1Pharmaceutical Sciences, Vancouver General Hospital, 855 West 12thdefined as primary or secondary [1–5]. A major compo-nent of SRB is outcome reporting bias (ORB), whichpublished, with estimated odds for publication beinto four times greater than those not reaching scance [6]. The distorted presentation of study outtion of a study analysis (e.g. subgroup), inconsistenciesin predefined measurement scales or time-points fordata collection, or the re-ranking of outcomes previouslypublication, indicating a high prevalence of SRB [3].There is also evidence to suggest that study outcomeswhich are statistically significant are more likely to beperformed in a study, may lead to the over- or underestimation of treatment effects or harms. Cochrane systematicreviews of interventions are required to assess the risk of SRB, achieved in part by applying the Cochrane risk of biastool to each included randomised trial. The Cochrane Handbook outlines strategies for a comprehensive risk of biasassessment, but the extent to which these are followed by Cochrane review groups (CRGs) has not been assessedto date. The objective of this study was to determine the methods which CRGs require of their authors to addressSRB within systematic reviews, and how SRB risk assessments are verified.Methods: A cross-sectional survey was developed and distributed electronically to the 52 CRGs involved inintervention reviews.Results: Responses from 42 CRGs show that the majority refer their authors to the Cochrane Handbook for specificinstruction regarding assessments of SRB. The handbook strategies remain variably enforced, with 57 % (24/42) ofCRGs not requiring review authors to search for included trial protocols and 31 % (13/42) not requiring that contactwith individual study authors be attempted. Only half (48 %, 20/42) of the groups consistently verify review authors’assessments of the risk of SRB to ensure completeness.Conclusions: A range of practices are used by CRGs for addressing SRB, with many steps outlined in the CochraneHandbook being encouraged but not required. The majority of CRGs do not consider their review authors to besufficiently competent to assess for SRB, yet risk of bias assessments are not always verified by editors beforepublication. The implications of SRB may not be fully appreciated by all CRGs, and resolving the identified issuesmay require an approach targeting several steps in the systematic review process.Keywords: Selective reporting bias, Outcome reporting bias, Systematic review, Cochrane CollaborationBackgroundSelective reporting bias (SRB) is a form of reporting biasin which certain components of conducted research areoccurs when only a subset of outcomes originally mea-sured in a study is selected for publication [2, 3].An analysis of randomised controlled trials (RCTs)bias: a survey of CochraneEmma K Reid1*, Aaron M Tejani2, Lawrence N Huan3, Greand Monisha Kabir6AbstractBackground: Selective reporting bias (SRB), the incomple© 2015 Reid et al. This is an Open Access artic(http://creativecommons.org/licenses/by/4.0),provided the original work is properly creditedcreativecommons.org/publicdomain/zero/1.0/selective reportingreview groupsry Egan1, Cait O’Sullivan4, Alain D Mayhew5publication of outcomes measured or of analysesle distributed under the terms of the Creative Commons Attribution Licensewhich permits unrestricted use, distribution, and reproduction in any medium,. The Creative Commons Public Domain Dedication waiver (http://) applies to the data made available in this article, unless otherwise stated.Reid et al. Systematic Reviews  (2015) 4:85 Page 2 of 8The impact of ORB in RCTs is inherently carried forwardwhen results are meta-analysed and included in systematicreviews [8]. In the ORBIT study, Kirkham et al. analysedthe prevalence of ORB within a cohort of Cochrane system-atic reviews and reported that 34 % of Cochrane reviews in-cluded one or more trials with a high suspicion for ORB inits analysis [2]. In an exploratory sensitivity analysis ofCochrane reviews with a single meta-analysis of the pri-mary outcome, close to one fifth of those which reportedstatistically significant results became non-significant onceadjusted for the presence of ORB in its primary studies [2].In addition, almost one quarter (23 %) of the results in thissubgroup with statistical significance would have exagger-ated the treatment effect by 20 % or more [2].The Cochrane Collaboration is an international net-work of researchers, health practitioners and patient ad-vocates, aiming to produce and publish credible andaccessible health information free from conflicts of inter-est. Cochrane reviews are recognised globally as beingsystematic reviews of the highest standard in evidence-based health care [9]. For Cochrane systematic reviews,the Methodological Expectations of Cochrane Interven-tion Reviews (MECIR) standards have been developed toensure high-quality and consistent review developmentand reporting [10]. All Cochrane reviews must include anassessment of risk of bias [10]. Cochrane review authorsare strongly encouraged to approach this assessment in adomain-based fashion by applying the Cochrane risk ofbias tool, which addresses several domains of bias (i.e. se-quence generation, allocation concealment, blinding ofparticipants and personnel, blinding of outcome assess-ment, incomplete outcome data, selective reporting) foreach trial considered for inclusion [4]. Instructions for per-forming the assessment of risk of bias are outlined in theCochrane Handbook for Systematic Reviews of Interven-tions, including factors to consider when determining if adomain should be judged as “low,” “high” or “unclear” riskof bias [4]. Briefly, review authors are asked to make thejudgement and include a description of how it was made,referencing a direct quote or describing the text fromwhich the judgement was concluded, and include a com-ment supporting the judgement. For the SRB domain, thehandbook provides instruction for the assessment of select-ive outcome reporting. It encourages review authors toconstruct a table or “matrix” of trials with their reportedoutcomes to better identify gaps in reported outcomes. Re-view authors are also encouraged to compare a trial’s proto-col or registry data to its final publication to assess fordiscrepancies in outcome reporting and to contact trial au-thors for unreported outcome data whenever possible [4].It is expected that review authors incorporate their riskof bias assessment into their final analysis. Judgementabout which bias domains have the greatest potential toaffect their particular review’s results is a criticalconsideration for the review author [4]. Strategies to ad-dress the implications of the risk of bias include present-ing multiple analyses of data accounting for different typesof bias (e.g. worst-case scenario analysis for missing pa-tient data), restricting the primary analysis to those studiesonly carrying a low (or unclear) risk of bias (e.g. sensitivityanalysis for blinded vs open label studies), or presentingall studies accompanied by a detailed narrative discussionof the risk of bias [4].The consideration of the review authors, editors andstaff from Cochrane review groups (CRGs) regarding thisassessment of risk of bias is crucial for the publicationof reliable, balanced and objective systematic reviews.Cochrane review authors, however, may not be reliably de-tecting ORB within trials [2]. Under-recognition of ORBhas recently been demonstrated in a study evaluating 30well-respected biomedical journals [11]. In this survey-based analysis, half of responding journals indicated thatORB was uncommonly or never detected in their journal’seditorial procedures and 64 % of the journals had nomethod at all in place for detecting ORB [11].Gaining knowledge of the understanding of and appre-ciation for SRB by Cochrane review groups and themethods they implement to account for the effect ofSRB from primary studies in systematic reviews will helpinform how current methods may be improved.The primary aim of this research was to identify thepractices of CRGs for minimising SRB in their system-atic reviews. This includes identifying the instructionprovided by CRGs to their review authors to assess forand address selective outcome reporting in the variousstages of a systematic review (its protocol, the final re-view, and subsequent updates). In addition, the specificmethods of each of the CRGs for verifying the complete-ness of the assessment of SRB would be determined. Asecondary component of the research was to explorehow fully CRGs believe their review authors understandand appreciate SRB.MethodsA cross-sectional survey with 21 questions (see Additionalfile 1) was developed with questions pertaining to thefollowing seven themes:(1) Instruction provided to authors regarding SRB(2) Consideration for SRB in systematic review protocol(3) Assessment of risk of SRB within the RCTs in asystematic review(4) Assessment of risk of SRB on the level of thesystematic review(5) Assessment of risk of SRB in updates of systematicreviews(6) Importance of SRB to review authors(7) GeneralReid et al. Systematic Reviews  (2015) 4:85 Page 3 of 8The majority (11/21) of questions addressed the thirdtheme (assessment of risk of SRB within the RCTs in asystematic review). The responses to more than half ofthe questions were categorical, with the number of cat-egories ranging from three to five. The option for free-text (“other” or “sometimes”) responses, however, wasalso available in almost all questions. The survey contentwas piloted for relevance with two active members ofthe Cochrane Bias Methods Group with knowledge ofsystematic reviews and SRB. Thereafter, three scientistswith no Cochrane involvement piloted the survey withregards to readability. The questions were further ad-justed based on these individuals’ feedback, to ensurerelevance and clarity.The survey was distributed to the 52 Cochrane reviewgroups who conduct reviews of clinical interventions ex-cluding only the Methodology Review Group. The man-aging and coordinating editor(s) in each group wereinvited via email to participate in the survey. These edi-tors could choose to respond themselves or delegate the re-sponse of the survey to another group member. Theintended respondent was an individual regularly involved inreviewing assessments of risk of bias in systematic reviewsor having sufficient knowledge of this process to accuratelyrepresent the practices of the CRG as a whole. Consultationbetween group members was encouraged so that the re-sponses would capture group practices rather than individ-ual variations. One survey per CRG was accepted.Distribution of the survey began in December 2013,and collection of survey submissions continued untilMarch 2014. Up to three reminder emails were sent atapproximately two-week intervals.Data were collected and tabulated with no identifyinginformation via the FluidSurveys™ survey platform. De-scriptive statistics were used to analyse survey responses.Five authors independently assessed all survey submis-sions, and a meeting was held to discuss the responses,including open-ended questions, to identify importantissues and patterns. The final decision on issues and pat-terns to discuss further was determined by consensus.Ethical approval for the study was received from boththe Behavioural Research Ethics Board at the Universityof British Columbia and the Fraser Health ServicesAuthority, Vancouver, Canada.ResultsOf the 52 CRGs invited to participate in the study, 42provided responses (81 %). In the one instance wheretwo completed surveys were received from one group,the submitting group members were contacted andasked to clarify via a single survey submission. Open-ended responses accounted for approximately one fifthof all responses. The meeting of five authors to discusssurvey responses identified trends in the open responses.Any key issues or patterns that were identified in cat-egorical and open survey questions will be discussed inthe categories below. Full results, including all openresponses, are available via Figshare (http://dx.doi.org/10.6084/m9.figshare.1421975).Instruction provided to authors regarding SRBThe majority of CRGs refer their review authors to theCochrane Handbook for instruction regarding determiningoutcomes for analysis (86 % [36/42]), selecting trials for in-clusion (78 % [33/42]) and conducting their assessment ofSRB (86 % [36/42]). There was also a trend for referring re-view authors to group-specific instruction or guidelines.Consideration for SRB in systematic review protocolAlmost all CRGs require a statement in their systematicreview protocols that the Cochrane risk of bias tool willbe applied to all included RCTs (88 % [37/42]). Only38 % (16/42) require a description within the protocol ofhow the assessment of the risk of SRB will be incorpo-rated into the results and/or discussion of the review.Assessment of risk of SRB within the RCTs in asystematic reviewThree quarters of CRGs stated that review authors areresponsible for performing assessments of risk of SRB(76 % [32/42]). Editorial staff members also play a rolein performing SRB assessments, with managing editorinvolvement reported in 50 % [21/42] of CRGs. Someopen responses indicated methodological and statisticaleditors perform assessments of SRB.Less than half of CRGs report they always review riskof bias assessments in submitted systematic reviews(48 % [20/42]). Nineteen percent (8/42) of CRGs reportnever verifying risk of bias assessments for SRB specific-ally or verifying only that the risk of bias tool is completed.It appeared from open responses that risk of bias assess-ments were more likely to be verified if the risk of biastool appeared to be filled out incorrectly by review au-thors. CRGs reported a very wide range (from 0–100 %)for the number of reviews with SRB assessments that re-quire revision before publication. The most frequently re-ported requirement for revision was the risk of bias toolbeing incomplete or discrepant.Fifty-seven percent (24/42) of CRGs do not require re-view authors to seek out trial protocols as a step in per-forming their risk of SRB assessment. The time requiredto perform the search and lack of availability of proto-cols were indicated as barriers to performing this task.Additionally, 31 % (13/42) of CRGs do not require re-view authors to attempt to contact trial authors regard-ing the completeness of outcome data. In some cases,this step was reported to be encouraged, but not re-quired. The creation of a “matrix” of reported trialfeel the protocol search is a step that should be mandatoryReid et al. Systematic Reviews  (2015) 4:85 Page 4 of 8outcomes amongst trials included in the systematic reviewwas only reported by 14 % of CRGs (6/42), though a fewadditional groups indicated that data extraction forms orother group-specific tables may be used for this purpose.The incorporation of an assessment of the risk of SRBinto the results or discussion section of the systematicreview is required by 45 % (19/42) of CRGs. If CRGs re-ported that the incorporation is “sometimes” required, itwas typically only if SRB had been identified within thereview but not when SRB assessments were done anddid not find evidence of SRB.Assessment of risk of SRB on the level of the systematicreviewFifty-two percent (22/42) of CRGs report consistentlyassessing for the selective inclusion of RCTs in a finalsystematic review (i.e. is the subset of data includedfully representative of available trial data?). The system-atic review protocol is compared to the final reviewfor consistency in defined outcome analyses by 86 %(36/42) of CRGs. Eighty-six percent (36/42) of CRGsreported that if a discrepancy in a protocol-defined out-come is recognised, a justification for the change is in-cluded in the methods (or more specifically, the“Differences between protocol and review”) section ofthe systematic review.Assessment of risk of SRB in updates of systematic reviewsDespite being considered a mandatory MECIR standard,only three quarters (31/42) of CRGs reported insistingthat the Cochrane risk of bias tool be applied to allRCTs included when performing an update of a system-atic review. In several instances, it was reported thatonly newly incorporated RCTs were subject to assess-ment with the risk of bias tool.Importance of SRB to review authorsCRGs were asked to classify their authors’ degree of un-derstanding of the existence of SRB, the implications ofSRB, their competency in completing and motivationfor completing assessments of SRB (see Fig. 1). Authorswere perceived as having a greater degree of under-standing of the existence of SRB than the implications ofSRB (45 vs 26 % had a moderate or large extent of un-derstanding, respectively). Thirty-one percent of groupsreported their authors had a “little” degree of compe-tency in performing SRB assessments. Only one CRGrated their review authors as having a “large” degree ofmotivation to complete SRB assessments.When considering only those CRGs who reported thatthey refer their review authors to the Cochrane Handbookfor instruction for assessing SRB, only 23 % (9/39) classi-fied their review authors to be moderately to largelycompetent in performing SRB assessments.in all systematic reviews, despite its additional timerequirement.The Cochrane Handbook recommends that authors oftrials be contacted in attempt to access all pre-specifiedoutcome data in the case of discrepancies [4], yet oursurvey suggests almost one third of CRGs make no at-tempt to do so. We also consider this step critical toGeneralFour of 42 CRGs reported having frequently contactedthe Cochrane Bias Methods Group for guidance in theprevious year. Over 60 % of CRGs report never havingcontacted the Cochrane Bias Methods Group inthis time.DiscussionOur survey in full provides an abundance of informa-tion. Many of the results identify issues regarding the ex-tent to which SRB is recognised and appreciated byCRGs. The focus of this discussion is on the issues forwhich we could identify a possible course of action forimprovement.The completeness of SRB assessments in systematicreviews relies on a comprehensive evaluation of thepresence of bias in included RCTs. Despite the CochraneHandbook being the primary resource for CRGs forinstructing risk of bias assessments, its recommenda-tions for evaluating SRB are implemented to a variableextent by different groups.An essential step in assessing SRB is searching for eachtrial’s protocol and comparing the originally defined out-comes with those in the final publication for complete-ness. It is reasonable to hypothesise that, with 57 % ofCRGs not requiring that trial protocols be identified andreviewed, this disparity contributes to frequent “unclear”risk of bias assessments in the SRB domain. As time andresources involved in protocol and trial registry searchesare identified barriers to completing the step, it would beadvantageous to increase the guidance in the CochraneHandbook regarding streamlining these searches. Agreater availability of trial protocols in the public domainwould also help alleviate this strain [12]. Fortunately, theuse of trial registries has increased since the introductionof the first trial registry platform (i.e. ClinicalTrials.gov) in2000, and the implementation of policies encouraging theregistration of trial design information prior to patientenrolment by the International Committee of MedicineJournal Editors (ICMJE) and the United States Food andDrug Administration (US FDA) in 2005 [13] and 2007respectively [14]. Given the important implications of SRBand the existing resources to access many protocols, wemaking an informed judgement regarding the bias riskfor SRB.?Reid et al. Systematic Reviews  (2015) 4:85 Page 5 of 8After an assessment of the risk of SRB has been made,it is prudent to incorporate these findings into the re-sults and discussion of the systematic review in the man-ner defined in the review protocol. The incorporation ofFig. 1 Q20: How do you rate your review authors’ understanding of SRBthese assessments was reported to be a mandatory stepbefore review publication by 45 % of CRGs. Somegroups indicated that a comment on SRB would bemade only if the bias was recognised to be present. Con-sidering the potentially extensive implications of SRB onthe findings of any trial, we feel striving to provide asmuch detail as possible around its presence or absenceshould be a priority in all systematic reviews to max-imally inform readers.A lack of understanding and appreciation for the im-plications of SRB, as well as how one may comprehen-sively assess for it, may very well be the root of theaforementioned issues. In a survey administered bySavovic and colleagues, SRB was identified by Cochranereview authors as being one of the most difficult types ofbias to analyse in risk of bias assessments [15]. Further,our survey included a set of questions inquiring, in gen-eral, how well-equipped each CRG believes their reviewauthors are in addressing SRB in their reviews. Only24 % of groups consider their authors to be moderatelyor largely capable of performing SRB assessments, whichsuggests a perceived knowledge gap for the review au-thors. Despite this, only 48 % of CRGs reported that as-sessments of risk of SRB in their reviews are consistently(“always”) verified before publication, suggesting this is adiscrepancy worth addressing.We feel the approach to addressing the SRB-related is-sues recognised in our survey would require targetingseveral steps in the systematic review process. We fullyacknowledge that our survey did not address the trainingand skills of review authors directly and focused insteadon editorial staff. It remains prudent, however, to includereview authors in our proposed approach to minimisethe impact of SRB in systematic reviews.We suggest starting from the beginning, during thetraining of Cochrane review authors. We propose thatstandard author training be structured in a way that in-cludes a focused review of SRB, including a focus on itsimplications and the precise steps for its proper assess-ment. Education could also go beyond this initial train-ing, with the introduction of online learning modules orrefresher courses for assessing risk of bias for both newand more seasoned review authors.For the assessment of risk of SRB itself, the steps out-lined in the Cochrane Handbook should more consistentlybe followed. Optimal mandatory steps would require edi-torial staff to enforce the protocol or trial registry searchand the contact attempts with original trial authors. Bydoing this, trial exclusion due to the lack of relevant out-come data as a result of SRB could be minimised.The handbook-described organisation of trials andtheir reported outcomes in a matrix was identified to bean underused strategy. One suggestion that may help re-view authors take a systematic approach to assessingSRB is by each CRG implementing a standardisedmatrix-based data extraction form (Fig. 2). On this form,n et al. [16])Reid et al. Systematic Reviews  (2015) 4:85 Page 6 of 8review authors would not only record the trial outcomesas described previously [4, 16] but would also documentboth the protocol search and author contact attempts.This would help ensure each of these steps is performedand would make the subsequent justification for the riskof bias assessment very transparent. The proposed matrixform outlines actions that need to take place in a checklistformat. This format could increase clarity for review au-thors and facilitate the editorial process for CRGs by pre-senting the data in an intuitive and concise manner. Thiswould allow CRG editorial staff to quickly assess andverify the work of the review authors in assessing SRB.Fig. 2 Proposed matrix-based data extraction form. (Adapted from DwaTo help clarify the approach of review authors for mak-ing final risk of bias judgements for SRB, we suggest an al-gorithm that lays out each step in the process (Fig. 3). Ifthe protocol of an included study is not available, for ex-ample, the ensuing step is to check trial registries to locateoutcome data. If no such data are found, an attempt tocontact study authors is made. If the protocol or initialoutcome data are provided, a comparison to final reportedoutcomes is made and the consistency determines “high”or “low” risk of SRB. If no response is received from thestudy authors, an “unclear” judgement is selected.In order to verify the quality of risk of bias assess-ments for SRB in a given systematic review, a specifiedmember of editorial staff could be delegated to double-check the primary work. This could be performed as arandom sample of a certain percentage, e.g. 10 %, of in-cluded trials, as a means for quality assurance.Strengths and limitationsThe high survey response rate (81 %) is an importantstrength of our study. Respondents were intended to beCRG staff members or authors with personal involvementor familiarity of risk of bias assessment practices withintheir group, with discussion amongst members encour-aged. Collecting only a single survey submission from eachCRG, however, may limit the scope of the informationprovided to an individual viewpoint rather than reflect thepractices of all authors. Additionally, the skill level of re-view authors themselves for addressing SRB was not dir-ectly assessed and instead was reported only as how it wasperceived by the survey respondent. This would be betterinvestigated in a secondary study questioning reviewauthors and sampling their reviews directly.Fig. 3 Proposed algorithm for judging risk of selective reporting biasin primary studiesFrequency and reasons for outcome reporting bias in clinical trials:http://www.cochrane.org/about-us. Accessed 19 May 2015.Reid et al. Systematic Reviews  (2015) 4:85 Page 7 of 8The use of mainly categorical responses for our surveyquestions may limit the interpretation of the full scopeof each response, but the option to elaborate with afree-form response was used to minimise this. With anysurvey, there is the possibility of misinterpretation ofthe questions; piloting of our survey questions was per-formed to lessen this potential. The scope of our surveyis limited to a single component of the Cochrane risk ofbias tool, which only represents a small portion of thelabours of writing and updating Cochrane reviews [15].Future research is aimed at determining practices ofCochrane review authors themselves for assessing SRBwithin a sample of reviews, and also exploring otherbias domains.ApplicationThis study will be of relevance to Cochrane review andmethodology groups as well as non-Cochrane systematicreviewers. It will also provide insight for future updatesto the Cochrane risk of bias tool and the CochraneHandbook. It is of importance to the Strengthening theReporting of Observational studies in Epidemiology(STROBE) efforts [17], the use of the Prospective Regis-tering of Systematic Reviews (PROSPERO) [18], the devel-opment of core outcome sets for the Core OutcomeMeasures in Effectiveness Trials (COMET) Initiative [19],the use of the Consolidated Standards of Reporting Trials(CONSORT) Statement [20], and the Preferred ReportingItems for Systematic Reviews and Meta-Analyses(PRISMA) Statement [21].ConclusionsOur study indicates that the recommendations in theCochrane Handbook for assessing SRB are variablyenforced by CRGs. The majority of CRGs do not considertheir review authors sufficiently competent to assess forSRB, yet risk of bias assessments are not consistently veri-fied by editors before publication. The implications of SRBmay not be fully appreciated by all CRGs, and resolvingthe identified issues may require a multi-faceted approachtargeting several steps in the systematic review process.Additional fileAdditional file 1: Survey Questions. Copy of original surveyinstructions and questions for reference.AbbreviationsSRB: selective reporting bias; ORB: outcome reporting bias; RCTs: randomisedcontrolled trials; MECIR: Methodological Expectations of CochraneIntervention Reviews; CRG: Cochrane review group; ICMJE: InternationalCommittee of Medicine Journal Editors; US FDA: United States Food andDrug Administration; STROBE: Strengthening the Reporting of Observationalstudies in Epidemiology; PROSPERO: Prospective Registering of SystematicReviews; COMET: Core Outcome Measures in Effectiveness Trials;PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.10. Chandler J, Churchill R, Higgins J, Lasserson T, Tovey D. Methodologicalstandards for the conduct of new Cochrane Intervention Reviews. 2013:version 2.3. In: Methodological Expectations of Cochrane Interventioninterviews with trialists. BMJ. 2010;341:c7153. doi:10.1136/bmj.c7153.7. Hazell L, Shakir SA. Under-reporting of adverse drug reactions: a systematicreview. Drug Saf. 2006;29:385–96.8. Dwan K, Kirkham JJ, Williamson PR, Gamble C. Selective reporting ofoutcomes in randomized controlled trials in systematic reviews of cysticfibrosis. BMJ Open. 2013;3(6), e002709.9. The Cochrane Collaboration. Cochrane community: about us. 2014.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsER, AT, LH and GE contributed to the project concept, the content and thedesign of the survey. ER distributed and collected survey submissions. ER, AT,LH, GE and CO examined survey submissions and identified themes for themanuscript. ER, AM and MK drafted the manuscript and all authors contributedto its revision. All authors read and approved the final manuscript.AcknowledgementsThe authors would like to acknowledge Matthew Page for the guidance informulating survey concepts and questions as well as Jamie Kirkham, MaryMacCara, Anna-Jean Reid and Elizabeth Reid for piloting the survey andproviding feedback. We thank Toby Lasserson and David Tovey for theirsupport in the promotion of the survey via the Cochrane Editorial Bulletin.We thank the Canadian Institutes of Health Research (CIHR Funding ReferenceNumber—CON-105529) and the Ontario Ministry of Health and Long-TermCare for their financial support of the Cochrane Methods—Bias Group for thecontribution of AM and MK. Finally, we thank the CO-OP Program at Universityof Ottawa for financial support for the contribution of MK.Author details1Pharmaceutical Sciences, Vancouver General Hospital, 855 West 12thAvenue, Vancouver, BC V5Z 1M9, Canada. 2Therapeutics Initiative (Faculty ofMedicine), University of British Columbia, 2176 Health Sciences Mall,Vancouver, BC V6T 1Z3, Canada. 3Department of Pharmacy, RichmondGeneral Hospital, 7000 Westminster Highway, Richmond, BC V6X 1A2,Canada. 4Island Health Clinical Pharmacy Programs, 375 Second Avenue,Campbell River, BC V9W 3V1, Canada. 5Knowledge Synthesis Group,Cochrane Methods—Bias, Ottawa Hospital Research Institute, 501 SmythRoad, Ottawa, ON K1H 8L6, Canada. 6Department of Biology, University ofOttawa, 75 Laurier Ave E, Ottawa, ON K1N 6N5, Canada.Received: 27 January 2015 Accepted: 2 June 2015References1. Hutton JL, Williamson PR. Bias in meta-analysis due to outcome variableselection within studies. Appl Stat. 2000;49(3):359–70.2. Kirkham JJ, Dwan KM, Atlman DG, Gamble C, Dodd S, Smyth R, et al. Theimpact of outcome reporting bias in randomized controlled trials on acohort of systematic reviews. BMJ. 2010;340:c365. doi:10.1136/bmj.c365.3. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al.Systematic review of the empirical evidence of study publication bias andoutcome reporting bias. PLoS One. 2008;3(8), e3081.4. Higgins JPT, Altman DG, Sterne JAC. Chapter 8: Assessing risk of bias inincluded studies. In: Higgins JPT, Green S, editors. Cochrane Handbook forSystematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration,2011. http://www.cochrane-handbook.org. Accessed 13 Dec 2014.5. Norris SL, Holmer HK, Fu R, Ogden LA, Viswanathan MS, Abou-Setta AM.Clinical trial registries are of minimal use for identifying selective outcomeand analysis reporting. Res Syn Meth. 2014;5:273–84.6. Smyth RM, Kirkham JJ, Jacoby A, Altman DG, Gamble C, Williamson PR.Reviews (MECIR). Cochrane Editorial Unit. 2013. http://editorial-unit.cochrane.org/sites/editorial-unit.cochrane.org/files/uploads/MECIR_conduct_standards%202.3%2002122013.pdf (2013). Accessed 19 Jan 2015.11. Huan LN, Tejani AM, Egan G. Biomedical journals lack a consistent methodto detect outcome reporting bias: a cross-sectional analysis. J Clin PharmTher. 2014;39(5):501–6. doi:10.1111/jcpt.12172.12. Chan A, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al.SPIRIT 2013 statement: defining standard protocol items for clinical trials. AnnIntern Med. 2013;158:200–7. doi:10.7326/0003-4819-158-3-201302050-00583.13. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinicaltrial registration: a statement from the International Committee of MedicalJournal Editors. Ann Intern Med. 2004;141:477–8.14. U.S. Food and Drug Administration: Food and Drug AdministrationAmendments Act (FDAAA) of 2007. http://www.fda.gov/regulatoryinformation/legislation/federalfooddrugandcosmeticactfdcact/significantamendmentstothefdcact/foodanddrugadministrationamendmentsactof2007/default.htm (2007) Accessed 17 Jan 2015.15. Savovic J, Weeks L, Sterne JAC, Turner L, Altman DG, Moher D, et al.Evaluation of the Cochrane Collaboration’s tool for assessing the risk of biasin randomized trials: focus groups, online survey, proposedrecommendations and their implementation. Syst Rev. 2014;3:37.doi:10.1186/2046-4053-3-37.16. Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C,Williamson PR. Assessing the potential for outcome reporting bias in areview: a tutorial. Trials. 2010;11:52.17. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbrouke JP.The Strengthening the Reporting of Observational Studies in Epidemiology(STROBE) statement: guidelines for reporting observational studies.PLoS Med. 2007;4(10), e296. doi:10.1371/journal.pmed.0040296.18. Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. Thenuts and bolts of PROSPERO: an international prospective register ofsystematic reviews. Syst Rev. 2012;1:2. doi:10.1186/2046-4053-1-2.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionReid et al. Systematic Reviews  (2015) 4:85 Page 8 of 819. Williamson PR, Clarke M. The COMET (Core Outcome Measures inEffectiveness Trials) Initiative: its role in improving Cochrane Reviews [editorial].Cochrane Database Syst Rev. 2012. doi:10.1002/14651858.ED000041.20. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updatedguidelines for reporting parallel group randomized trials. Ann Intern Med.2010;152(11):726–32. doi:10.7326/0003-4819-152-11-201006010-00232.21. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items forsystematic reviews and meta-analyses: the PRISMA statement. BMJ.2009;339:b2535. doi:10.1136/bmj.b2535.Submit your manuscript at www.biomedcentral.com/submit


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items