UBC Faculty Research and Publications

Does journal endorsement of reporting guidelines influence the completeness of reporting of health research?… Shamseer, Larissa; Stevens, Adrienne; Skidmore, Becky; Turner, Lucy; Altman, Douglas G; Hirst, Allison; Hoey, John; Palepu, Anita; Simera, Iveta; Schulz, Kenneth; Moher, David May 24, 2012

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13643_2012_Article_33.pdf [ 215.8kB ]
JSON: 52383-1.0223320.json
JSON-LD: 52383-1.0223320-ld.json
RDF/XML (Pretty): 52383-1.0223320-rdf.xml
RDF/JSON: 52383-1.0223320-rdf.json
Turtle: 52383-1.0223320-turtle.txt
N-Triples: 52383-1.0223320-rdf-ntriples.txt
Original Record: 52383-1.0223320-source.json
Full Text

Full Text

PROTOCOL Open AccessDoes journal endorsement of reporting guidelinesinfluence the completeness of reporting of healthresearch? A systematic review protocolLarissa Shamseer1, Adrienne Stevens1, Becky Skidmore2, Lucy Turner1, Douglas G. Altman3, Allison Hirst3,John Hoey4, Anita Palepu5, Iveta Simera3, Kenneth Schulz6 and David Moher1*AbstractBackground: Reporting of health research is often inadequate and incomplete. Complete and transparentreporting is imperative to enable readers to assess the validity of research findings for use in healthcare and policydecision-making. To this end, many guidelines, aimed at improving the quality of health research reports, havebeen developed for reporting a variety of research types. Despite efforts, many reporting guidelines are underused.In order to increase their uptake, evidence of their effectiveness is important and will provide authors, peerreviewers and editors with an important resource for use and implementation of pertinent guidance. The objectiveof this study was to assess whether endorsement of reporting guidelines by journals influences the completenessof reporting of health studies.Methods: Guidelines providing a minimum set of items to guide authors in reporting a specific type of research,developed with explicit methodology, and using a consensus process will be identified from an earlier systematicreview and from the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network’s reportingguidelines library. MEDLINE, EMBASE, the Cochrane Methodology Register and Scopus will be searched forevaluations of those reporting guidelines; relevant evaluations from the recently conducted CONSORT systematicreview will also be included. Single data extraction with 10% verification of study characteristics, 20% of outcomesand complete verification of aspects of study validity will be carried out. We will include evaluations of reportingguidelines that assess the completeness of reporting: (1) before and after journal endorsement, and/or (2) betweenendorsing and non-endorsing journals. For a given guideline, analyses will be conducted for individual and thetotal sum of items. When possible, standard, pooled effects with 99% confidence intervals using random effectsmodels will be calculated.Discussion: Evidence on which guidelines have been evaluated and which are associated with improvedcompleteness of reporting is important for various stakeholders, including editors who consider which guidelines toendorse in their journal editorial policies.Keywords: Reporting guidelines, Evaluation, Systematic review, Completeness of reporting* Correspondence: dmoher@ohri.ca1Centre for Practice-Changing Research, Ottawa Hospital Research Institute,The Ottawa Hospital – General Campus, 501 Smyth Road, Box 201B, Ottawa,ON K1H 8L6, CanadaFull list of author information is available at the end of the article© 2012 Shamseer et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly cited.Shamseer et al. Systematic Reviews 2012, 1:24http://www.systematicreviewsjournal.com/content/1/1/24BackgroundReporting guidelinesReporting of health research is, in general, inadequate[1-6]. Complete and transparent reporting is imperativeto assessing the validity of reported treatment effectsand other findings of health research. A study’s methodsshould be described in enough detail so they can bereplicated, the analyses should follow the protocol, andthe results should be given in full enough detail to beincorporated into future research. Complete and trans-parent reporting enables clinicians and others to makebetter, more informed health care decisions. Transparentreporting is an integral part of the research process andfacilitates the interpretation of whether good science wasemployed. For instance, without a description of themethods used to control internal validity (for example,randomization, blinding) and external validity (for ex-ample, definition of the population under study), thereader is left to guess at whether the treatment effect es-timate reported is accurate. To ameliorate the problemof inadequate reporting, many guidelines have beendeveloped, aimed at improving the quality of reports ofhealth research. A reporting guideline aids authors inthe reporting of specific types of research and may beaccompanied by a checklist of important items to bereported and, potentially, a flow diagram to describe thestudy process or explicit guidance text [7,8].The EQUATOR (Enhancing the QUAlity of Transpar-ency Of health Research) Network is an internationalinitiative promoting transparent and accurate reportingof health-related research. As of October 2011, 191reporting guidelines were indexed in the Library forHealth Research Reporting on the EQUATOR website[9]. Of the 90 indexed in September 2009, 81 wereincluded in a systematic review that characterized theirdevelopment process. Guidelines in that review datefrom as early as 1986 but many are more recent; ap-proximately half were developed between 2005 and 2009[8]. Those numbers demonstrate considerable and in-creasing investment in the development of reportingguidelines.Importance for journal editorsJournals are the most important conduit for publishinghealth research. Some reporting guidelines have receivedpositive attention, in the form of endorsement by healthjournals (for example, the CONsolidated Standards OfReporting Trials (CONSORT) Statement is endorsed byover 600 journals). Such endorsement is typically evi-denced by a statement in a journal’s “Instructions toAuthors” regarding the use (suggested or required) ofone or more guidelines while preparing a study manu-script. Some journals publish editorials indicating theirsupport, while others institute mandatory submission ofa guideline checklist and/or flow diagram along withmanuscript submission.Editors are constantly striving to ensure that what ispublished in their journals is, clear, complete, transpar-ent, and as free from bias as possible. A recent surveyindicates that almost half of journal editors whoresponded (n = 67) consider completeness of reportingto be one of the top three factors when making a publi-cation decision [10]. In the same survey, however, 18%of the editors who were interviewed perceived endorse-ment of a reporting guideline as burdensome. Further-more, in an effort to uphold high standards, journaleditors may feel the need to endorse multiple reportingguidelines without knowledge of their rigor or ability toimprove reporting. Evidence about their effect may pro-vide editors with the rationale for making decisionsabout which to endorse at their journals.Evaluation of reporting guidelinesTo date, there have been few evaluations of reportingguidelines. In many ways reporting guidelines are achecklist of important items that ought to be carriedout. In other fields such checklists have proven to be ofgreat benefit. For example, an evaluation of the WorldHealth Organization (WHO) Surgical Safety Checklistshowed that its implementation is associated with a 47%reduction in the rate of death and 36% reduction in in-patient complications after surgery [11]. Not only dothese numbers demonstrate the usefulness of a checklistin improving mortality and morbidity in surgery, theyare arguably the most fundamental piece of informationneeded to initiate change in surgical practice. Based onthese numbers, implementation of the WHO checklist istaking place on a global scale. Similarly, checklists forreporting research must be evaluated in order to provideevidence for knowledge users to make informed deci-sions about their implementation.With respect to reporting guideline evaluation, only17% of guideline developers report an intention to for-mally evaluate their guidelines; 7% indicate an explicitintention not to evaluate their guideline post-publication[8]. In a 2008 survey of developers of 30 reporting guide-lines, only 17% (n = 5) stated having formally evaluatedthe impact of their guidelines on the completeness ofreporting of research for which it was intended [12].These numbers demonstrate a gap between guideline de-velopment and quality control, leaving developmentefforts wasted if stakeholders are unable to judge theireffectiveness and, as such, subsequently unable to en-dorse a guideline.One reporting guideline that has been extensively eval-uated is the CONSORT Statement [13-17]. In 2006, asystematic review of CONSORT effectiveness identifiedeight studies evaluating its impact [18]. A 2012 updateShamseer et al. Systematic Reviews 2012, 1:24 Page 2 of 9http://www.systematicreviewsjournal.com/content/1/1/24of that review identified 42 additional evaluations[19,20]. Both found that endorsement of CONSORT byjournals is significantly associated with a higher fre-quency of completely reported trials, at least for someitems of the CONSORT Statement. Evaluations of otherguidelines, such as the Standards for the Reporting ofDiagnostic Accuracy (STARD) statement in 2003 [21]and the Standards for Reporting Interventions in ClinicalTrials of Acupuncture (STRICTA) Statement [22], areknown to exist but have not yet been systematicallyidentified and synthesized. It may be that these examplesindicate a fraction of evaluations that actually exist.Bringing together evaluations of reporting guidelines willprovide a resource for knowledge users to aid in thedecision-making process of whether or not to implementa reporting guideline, improve the uptake of reportingguidelines overall, and highlight areas of reporting stillleft to be evaluated.ObjectivesThe objective of this systematic review is to evaluatewhether journal endorsement of reporting guidelinesinfluences the completeness of the reported literature bycomparing:1. Completeness of reporting of studies published injournals endorsing reporting guidelines before andafter endorsement.2. Completeness of reporting of studies published injournals that have and have not endorsed reportingguidelines.MethodsCriteria for including studies in this reviewTypes of studiesIn order to identify evaluations of reporting guidelines,we first need to determine the set of reporting guidelinesfor which we will be seeking evaluations and subse-quently, on which to base the search.Eligibility of reporting guidelines of interest Reportingguidelines for health research, including clinical, basicscience and laboratory research, and written only inEnglish, (for feasibility), will be included.Eligibility criteria will be based on a recent systematicreview that identified and characterized the reportingguideline process for 81 reporting guidelines for healthresearch [8]. Potential reporting guidelines will beincluded if they contain explicit text to guide authorsin the reporting of a specific type of research (whichmay or may not be accompanied by a checklist ofreporting criteria and/or a flow diagram) and if theydescribe how the guideline was developed and howconsensus among the developers was obtained. Thiscriteria is intended to differentiate reporting guidelinesfrom other efforts in which guidance was not devel-oped using explicit methods, even if a checklist is pro-duced, such as the recent guidance for reportingmyeloma trials - described as ‘consensus’ in the title -but no methods of development or consensus wereotherwise described [23]. Selection criteria will also ex-clude guidance on the formatting of reports of healthresearch, such as found in some journals’ 'Instructionsto Authors’. Reporting guidelines for purposes otherthan reporting research (for example, clinical case re-port forms) will also be excluded.For the purpose of the review, consensus is defined aselectronic communication, an online or in-person meet-ing, a group teleconference, or the use of a Delphiprocess among guideline developers. If authors providelittle to no detail about the method of consensus usedbut indicate the use of consensus, it will be included, aslong as other methods of development are explicitlydescribed.Where a reporting guideline has been updated overtime and several versions exist, each version will beincluded as a separate reporting guideline in order totrack the number of evaluations of each.Eligibility of evaluations of reporting guidelines Anycomparative study with the primary intent of assessingcompleteness of reporting of at least one study publishedin a journal that endorses reporting guidelines will beincluded and considered an “evaluation” for the remain-der; studies contained within an evaluation will betermed “studies”. For the first comparison, evaluationsusing before-after designs to compare completeness ofreporting of studies published pre- and post-endorsement within a given journal, or cohort of jour-nals, will be included. For the second comparison,evaluations using a cross-sectional design to comparecompleteness of reporting between studies published inendorsing and non-endorsing journals will be included.Evaluations examining both types of comparisons will beincluded.An editorial statement regarding the use of one ormore reporting guidelines is the minimum criterion fora journal to be considered an “endorser”, implying that,at least in principle, the guideline(s) is/are incorporatedinto the journal’s editorial process. When included eva-luations do not report the endorsement status of jour-nals publishing evaluated studies (that is, it is unclearwhether studies being evaluated were published in en-dorsing or non-endorsing journals), correspondingauthors will be contacted up to two times to provide thisinformation. If authors did not collect this information,they will be asked to provide the list of studies or jour-nals included in their evaluation; if feasible, members ofShamseer et al. Systematic Reviews 2012, 1:24 Page 3 of 9http://www.systematicreviewsjournal.com/content/1/1/24the review team will seek out the endorsement status ofincluded journals’ from “Instructions to Authors” ontheir websites. If not feasible to follow-up on an author’slist of studies or journals to obtain information on en-dorsement status, the evaluation will be excluded withthe reason provided. Further to endorsement status, thedate of a journal’s endorsement of a reporting guidelineis important to ascertain and will help determinewhether studies in a given journal were published after asufficient period of time to allow for changes in editorialpolicies to be realized in its publication output. For thisreview, six months will be considered a reasonableperiod. As it is unlikely that the date of journal endorse-ment will be reported in evaluations, if feasible, corre-sponding authors of evaluations or editors of evaluatedjournals will be contacted for this information. If un-available, evaluations will not be excluded on this basis;the most recent endorsement status will be used as asurrogate.Evaluations using interrupted time series designs willbe excluded.Since the CONSORT Statement is a known reportingguideline and a systematic review of its evaluations hasrecently been updated [20], data from eligible studieswill be included in this review. The search strategy ofthis review intends to identify any CONSORT evalua-tions published since the date of the last search of thatsystematic review (March 2010).Types of outcomesPrimary outcomeThe primary outcome will be completeness of reporting,This will be measured as adequate or inadequate report-ing of any of the following surrogate outcomes, asjudged and reported in included evaluations: one ormore items in a reporting guideline checklist; a “sum-mary score” of some or all checklist items; a flow dia-gram (if not already included as a checklist item); ornarrative guidance found in the text.Secondary outcomes1. Methodological/reporting quality of an evaluation’sincluded studies, as measured by any means in theevaluation (for example, Jadad scale for randomizedcontrolled trials).2. Any unwanted effect described as associated withuse of a reporting guideline (for example, increasedword count).Methods for searching for studiesIdentifying reporting guidelines of interestReporting guidelines included in the Moher 2011 sys-tematic review [8], identified through a search ending inSeptember 2009, will be automatically included in thisreview; guidelines were selected using the same eligibilitycriteria. From October 2009 to June 2011, reportingguidelines identified by the EQUATOR Network througha comprehensive PubMed search (see Additional file 1:Appendix 1 for full strategy) will be screened for poten-tial inclusion (that is, screened using the above-listedcriteria).Identifying evaluations of reporting guidelinesA comprehensive two-phase approach has been devel-oped to identify evaluations of reporting guidelines.1. Some existing reporting guidelines are formallynamed using a unique acronym in their title (forexample, PRISMA: Preferred Reporting Items forSystematic reviews and Meta-Analyses). As well,some reporting guidelines have been widelydisseminated across multiple journals or areaccompanied by explanatory documents (forexample, CONSORT). To capture evaluations forsuch guidelines, a strategy to identify evaluationsreferring to any publication of a uniquely namedreporting guideline was developed. MEDLINE,EMBASE, and the Cochrane Methodology Registerwill be searched from 1990 onward using a searchstrategy that includes the unique acronym of eachreporting guideline (see MEDLINE strategy inAdditional file 2: Appendix 2). This search strategyis largely based on that used to identify evaluationsof the CONSORT Statement in the aforementionedCONSORT systematic review [20].2. For other reporting guidelines with commonly usedacronyms or those without acronyms, a differentapproach has been developed. For instance,“TREND” refers to guidance on the TransparentReporting of Evaluations with NonrandomizedDesigns but may also refer to statistical trends.Potential evaluations of such guidelines will beidentified through forward citation searching of eachinstance of a guideline’s publication in the Webcitation index, Scopus. For those guidelines,MEDLINE and EMBASE will be searched to identifyadditional, existing multiple publications to search inScopus.When a reporting guideline has been updated overtime and several versions exist (for example, 1996, 2001and 2010 versions of CONSORT), potential evaluationsof all versions will be accounted for in the search.Reporting guideline developers will also be contacted toascertain knowledge of any unpublished or in-progressevaluations. Reference lists of related systematic reviewsencountered during the screening process will also beShamseer et al. Systematic Reviews 2012, 1:24 Page 4 of 9http://www.systematicreviewsjournal.com/content/1/1/24hand-searched to identify additional evaluations. Anyevaluations detected by members of our team, publicallyavailable before the date of the last search will be con-sidered for inclusion.Published and unpublished evaluations will beincluded. Letters, comments and editorial publicationtypes will be excluded. There will be no language restric-tion on the search strategy, but only those available inEnglish and French will be included, due to limitedtime and resources. Potential evaluations identified inother languages will be listed and set aside for consider-ation at a future date. Electronic searches were peerreviewed using the Peer Review of Electronic SearchStrategies (PRESS) Statement [24]; see Additional file 3:Appendix 3 for peer review comments.Data collection and analysisData managementFollowing the execution of all searches, the identifiedrecords (titles and/or available abstracts) will be collatedin a Reference ManagerW [25] database for de-duplica-tion. The final unique record set and full-text ofpotentially eligible studies will be exported to anInternet-based software, DistillerSRW (Evidence Partners,Ottawa, ON, Canada), through which screening ofrecords will be carried out. Extraction of data from stud-ies will be carried out in Microsoft Excel (2007, Micro-soft Corporation, Redmond, WA, USA).Study selectionSelecting reporting guidelines of interestTwo members of the research team will independentlyapply inclusion criteria to full text reports of potentialreporting guidelines. Screening forms will be pilotedusing a subset of records. Discordance betweenreviewers will be resolved through consensus or by athird team member.Selecting evaluations of reporting guidelinesThe results of the literature search will be assessed usinga two-step process:1. One individual will screen citations by titles and/orabstract according to the pre-specified screeningquestions (level 1). For those records deemed to be‘included’ and/or ‘unclear’, they will automaticallypass to the next level of screening (level 2).However, if the record is deemed ‘excluded’, then itwill be assessed by a second reviewer to confirmexclusion. This process is referred to as ‘liberalaccelerated’ screening, a more efficient means ofinitially assessing records for relevancy.2. Full-text screening will be conducted by twoindependent reviewers over two phases, where pre-defined questions will be split to create two levels(levels 2 and 3). This will be done in order toexpedite the screening process - eligibility based onsome criteria can be determined simply from theevaluation report whereas other criteria may requirecontact with corresponding authors of evaluations inorder to judge eligibility (for example, endorsementstatus of journals in the evaluations). Discordancebetween reviewers will be resolved throughconsensus or by a third team member.3. All screening forms will be piloted using a subset ofrecords. Screeners will not be blinded to studyauthors or journal of publication.Data extractionSeparate data extraction forms will be developed to cap-ture information needed for synthesis for each of thetwo comparisons of this review and will be piloted usinga subset of included evaluations and modified, asneeded. One reviewer will extract general study charac-teristics of included evaluations, with verification of arandom 10% of studies carried out by a second reviewer.Data on completeness of reporting for each reportingguideline will be extracted by one reviewer; a second re-viewer will verify the accuracy of the data from a ran-dom 20% sample of included evaluations. Anydiscrepancies between reviewers will be resolved by con-sensus or by a third member of the research team. Ifthere are greater than 50% discrepancies, 100% data veri-fication will be considered.Data items that will be extracted from evaluations willinclude: Name of reporting guideline being evaluated (withversion if applicable) Whether an included reporting guideline is anextension of the primary reporting guideline. If so,whether it is an official extension (that is, developedin collaboration with primary guideline authors) willbe determined. Collaboration will be defined asinclusion of at least one lead author of the originalguideline on the extension authorship list. Study design of the evaluation (for example, cross-sectional, cohort, and so on). General characteristics of the evaluation: first authorname; year of publication; country of correspondingauthor; corresponding author email address; sourceof funding. Characteristics of studies assessed in the evaluation:number of studies and publishing journals; date ofstudy publication; journal endorsement status anddate of journal endorsement (if available); whetherdate of endorsement was determined by evaluationauthors at the time of evaluation, obtained directlyShamseer et al. Systematic Reviews 2012, 1:24 Page 5 of 9http://www.systematicreviewsjournal.com/content/1/1/24from journals, or determined by review authors(based on current/surrogate status listed in journals’“Instructions to Authors”; extent of endorsementaccording to pre-defined categories (see ‘Subgroupanalysis’ section); medical specialty; guidelinechecklist items assessed, if applicable. Completeness of reporting will be collected asmeasured by any of the following surrogateoutcomes: Adequacy of reporting of individual checklistitems or a combination of items into a summaryscore. It is anticipated that most studies will evaluatecompleteness of reporting in this manner,since 94% of reporting guidelines identified ina recent systematic review included a checklist[9]. Item-by-item extraction forms will bedeveloped for each checklist-based reportingguideline that is assessed in at least oneevaluation. Where evaluations present data on variationsof checklist items, this data will be collectedand presented in subgroup analyses (see‘Subgroup analysis’ section). Calculation of a summary score may bemisleading since items within a checklist arenot necessarily of equal importance. However,if completeness of reporting is assessed in thismanner in included evaluations, data will becollected and analyzed. Adequacy of reporting according to narrativeguidance found in the text of a guidelinedocument. For evaluations of reporting guidelines withouta checklist, methods of evaluation ofcompleteness of reporting and overallcompleteness of reporting, as reported, will becollected. Adequacy of reporting other measures ofcompleteness of reporting, if assessed inevaluations. Other measures of study quality reported in theevaluation, however measured (for example, Jadadscale for randomized controlled trials). Potential unwanted effects from using a reportingguideline, as reported by authors of an includedevaluation, however reported.Authors of potentially included evaluations will becontacted up to two times to obtain additional informa-tion, if needed, such as outcome data or data not avail-able in published reports.Assessment of validity of included studiesOne reviewer will independently assess internal validityfor each included evaluation; this will be verified by asecond reviewer. A set of criteria has been developed forthis review, based on concepts presented in the DataCollection Checklist developed by the Cochrane Effect-ive Practice and Organisation of Care Review Group[26], the Strengthening the Reporting of ObservationalStudies in Epidemiology (STROBE) Statement [27], theNewcastle-Ottawa Quality Assessment Scale [28] andthe Cochrane Risk of Bias Tool [29].Criteria for assessing validity1. Whether more than one person evaluatedcompleteness of reporting of included studies2. Whether the set of items to be evaluated (methods)is the same as those which were evaluated (results)3. Whether the intended set of data is completelyreported or provided4. Whether the search strategy used to identify studiesin the evaluation was appropriate to the questionbeing asked5. Whether included studies have a particularpublication frequency within included journals6. If no to the item 5, whether confoundingconsidered/accounted for in the evaluationEach criterion will be judged as yes (high validity), no(low validity), unclear (not reported) and not applicable;support for judgments will also be provided. As nomethods exist for synthesizing these data into a sum-mary judgment, results of validity assessment will bepresented in tabular format for reader interpretation.Measures of effectThe first comparison is the completeness of reportingwithin endorsing journals before and after endorsement,and the second is completeness of reporting between en-dorsing and non-endorsing journals in the same period.If data for both comparisons are available in a singleevaluation, the evaluation will be included in both com-parisons (that is studies are published in endorsing andnon-endorsing journals and the endorsing cohortincludes the time periods before and after endorsement).Within each evaluation, the proportion of studies ad-equately reporting one or more checklist items for agiven guideline will be collected. For guidelines contain-ing only text-based recommendations, if evaluationsShamseer et al. Systematic Reviews 2012, 1:24 Page 6 of 9http://www.systematicreviewsjournal.com/content/1/1/24exist, recommendations will be grouped into meaningfulpseudo-checklist items and adequacy of reported datawill be collected for each item. A relative risk (RR) and99% confidence intervals will be calculated for eachstudy; a 99% confidence interval will ensure conservativeestimates of precision are obtained. A RR> 1.0 will indi-cate a higher proportion of studies adequately reportinga given checklist item.Likewise, means and standard deviations (SDs) will becollected for checklist items combined into a summaryscore; when medians and ranges are reported instead ofmeans and SDs, suitable approximations will be used, asdiscussed in the Cochrane Handbook [30]. A standar-dized mean difference (SMD) and 99% confidence inter-val will be calculated for each study; an SMD>0 willindicate more adequate reporting of checklist items con-tained within the total summary score.Information on the methodological quality of studieswithin evaluations and any unwanted effects of reportingguideline use will be presented as reported in the evalua-tions. No judgments will be made on these items; theywill be presented as collected from included evaluations.Unit of analysisThe unit of analysis in this review is an included evalu-ation. Within each evaluation, one or more includedstudies may be published in the same journal, therebynot independent due to a common effect from editorialpolicies. As such, assessment of this issue will be carriedout during validity assessment of evaluations, asdescribed above (see validity items 5 and 6).Dealing with missing dataCorresponding authors of potentially included evalua-tions will be contacted, up to two times, where data areneeded (that is endorsement status of include journals,completeness of reporting assessments). If the data arenot obtained and compromise the ability to include theevaluation in quantitative synthesis, they will beexcluded from the meta-analyses.Reporting BiasesAsymmetry of funnel plots is an established method forassessing the potential presence of publication bias intraditional systematic reviews of intervention effective-ness, subject to a sufficient number of included studies[30,31]. Funnel plots are a graphical representation of in-dividual study estimates of effect against a measure ofthe study’s size or precision. In the current study, thesample size is the number of studies included in eachevaluation. Although it is possible to generate funnelplots to assess the potential of publication bias withinour pool of included evaluations, both the suitability andpossible interpretation of such plots are unknown.Data synthesisFor each included evaluation we will present study charac-teristics, assessment of validity, and description of thereporting guideline evaluations in a series of tables and anarrative summary. Meta-analyses will be carried outusing the Review Manager [31] and Comprehensive Meta-analysis software [32]. If evaluations for each reportingguideline are similar enough on the basis of study design,for each outcome, effect estimates from each evaluationwill be pooled into a single, overall, effect estimate.Primary outcomePooled RRs and MDs with corresponding 99% confi-dence intervals, using a random effects model, will beused to compare completeness of reporting of studiesacross evaluations for each checklist item or summaryscore, respectively, for each guideline. Estimates of effectfor different reporting guidelines will not be pooled inany way; subgroups and totals will be provided for eachguideline, including various versions of a given reportingguideline, separately. Data from guidelines with onlytext-based recommendations will not be pooled due toanticipated variation in how adequacy of reporting datawas collected.Secondary outcomesIt is expected that methods of assessment of methodo-logical quality and reporting of unwanted effects will bevariable among included evaluations. When reported, adescriptive summary of methodological quality of studiesincluded in evaluations and unwanted effects, for eachreporting guideline, will be provided. No attempt will bemade to statistically synthesize these data.Subgroup analysisThe following subgroup analyses are planned, if feasible: Extent of reporting guideline endorsement by journalsincluded in evaluations: pre-defined groupingsdeveloped by authors of the recent CONSORTreview will be used [20]: (a) any editorial statementregarding use of a guideline; (b) recommendation ina journal’s “Instructions to Authors” to follow theguideline when preparing the manuscript; and(c) requirement for authors to submit guidelineadherence documentation (for example, completedCONSORT checklist) with their manuscript. Variations in checklist items: if variations in howevaluations report completeness of reporting fordifferent checklist items are encountered, data willbe presented in the main analyses according todifferent subgroups for each variation. For example,in the CONSORT systematic review, ‘blinding’ wasreported in four different ways among evaluationsShamseer et al. Systematic Reviews 2012, 1:24 Page 7 of 9http://www.systematicreviewsjournal.com/content/1/1/24and, therefore, data divided into four subgroups,accordingly, in the analysis [20]. Official and unofficial extensions of reportingguidelines: for reporting guidelines identified asextensions to a primary guideline, effect estimatesfor official and unofficial extensions will bepresented separately.Sensitivity analysesWe plan to conduct the following sensitivity analyses, ifpossible, to determine the influence on effect estimates: Six-month endorsement period. The primaryoutcome analysis will be restricted to evaluations ofstudies that were published at least six monthsfollowing the date of journal endorsement, for whichthe true date of endorsement could be obtained(that is, as provided in the report, by evaluationauthors or by journal editors, not the date on whichsurrogate status was obtained by review authors). Study outliers. Evaluations with effect estimatesoutside of the 99% CI of pooled RRs and SMDs willbe removed for sensitivity analysis.Assessment of heterogeneityWe plan to measure the inconsistency of study resultsusing the I2 heterogeneity statistic to determine the ex-tent of variation in effect estimates that is due to hetero-geneity rather than chance. Heterogeneity, as defined byHiggins, is measured as a percentage (%) where a value≤25% for I2 indicates low heterogeneity, 26% to 50% indi-cates moderate heterogeneity 51% to 75% indicates sub-stantial heterogeneity and 76% to 100% indicatesconsiderable heterogeneity [30]. Substantially heteroge-neous effect estimates will not be pooled. Possible rea-sons for heterogeneity will be explored in sensitivityanalyses; the pre-specified subgroup analyses, if feasible,will be examined to determine whether they provide pos-sible reasons for any observed statistical heterogeneity.Reporting of this reviewThis systematic review will be reported according to thePreferred Reporting Items for Systematic reviewsand Meta-Analyses (PRISMA) Statement [33] – a report-ing guideline for systematic reviews of healthcare inter-ventions – and will include a PRISMA checklist. Wherenecessary, we will adapt the reporting to ensure that allitems relevant to this review are included in the report.DiscussionThis systematic review aims to provide evidence to helpguide decision making for journal editors and publishers.While some editors are enthusiastic about reportingguidelines, it is likely a prudent policy to endorse andadhere to those reporting guidelines that are appropri-ately developed and provide some evidence of effective-ness; namely, that their use is associated with improvedcompleteness of reporting. The proposed systematic re-view will provide the evidence regarding reportingguideline effectiveness. As such we believe that we areproviding editors with evidence to help inform their pro-spective policy about specific reporting guidelines.Beyond journal editors we believe that the results of thisreview will be of relevance to other knowledge users,namely, peer reviewers and authors. There is an increasedinterest in asking peer reviewers to use reporting guide-lines as part of their assessment of manuscripts; 46% ofhealth journals surveyed (n= 116) mentioned reportingguidelines in their instructions to peer reviewers [34]. Theresults of this review will help peer reviewers decide whichreporting guidelines are effective and, thus, likely morebeneficial to use as part of the peer review process. Finally,authors should be more strongly encouraged to usereporting guidelines for which there is evidence that theyare associated with improved completeness of reporting.With information summarized in this review, publish-ers may be able to implement reporting guidelinesacross a wide spectrum of journals and reduce the bur-den on individual journals. Specifically, making reportingguidelines accessible and required at the time of submis-sion will greatly improve their use by authors and easethe burden of peer-reviewing by standardizing the selec-tion criteria across journals, specific to the research typ-ology being reviewed.Additional filesAdditional file 1: Appendix 1. EQUATOR network Pubmed ReportingGuidelines Search Strategy.Additional file 2: Appendix 2. MEDLINE search strategy for evaluationsof reporting guidelines with acronyms. Searches were tailored to searchEMBASE and the Cochrane Methodology Register. Searches for remainingreporting guidelines were conducted in Scopus.Additional file 3: Appendix 3. PRESS EBC Search Submission.AbbreviationsEQUATOR: Enhancing the Quallity and Transparency of health Research;CONSORT: Consolidated Standards of Reporting Trials; STARD: Standards forthe Reporting of Diagnostic Accuracy; STRICTA: Standards for ReportingInterventions in Clinical Trials of Acupuncture; PRISMA: Preferred ReportingItems for Systematic reviews and Meta-Analyses; PRESS: Peer Review ofElectronic Search Strategies; STROBE: Strengthening the Reporting ofObservational Studies in Epidemiology; RR: Relative risk; SMD: Standardizedmean difference.Competing interestsProfessor Altman, and Drs. Hoey, Moher and Schulz are executive membersof the EQUATOR network; Dr. Iveta Simera and Allison Hirst are EQUATORstaff members. The EQUATOR Network is funded by the National HealthService (NHS) National Library of Health, NHS National Institute for HealthResearch, NHS National Knowledge service, UK Medical Research Council,Canadian Institutes of Health Research, Scottish Chief Scientist Office, PanAmerican Health Organization.Shamseer et al. Systematic Reviews 2012, 1:24 Page 8 of 9http://www.systematicreviewsjournal.com/content/1/1/24This study is supported by a grant from the Canadian Institutes of HealthResearch (#234489). CIHR had no role in study design, plans for datacollection and analysis, decision to publish, or preparation of this protocol.Professor Altman is supported by Cancer Research UK, Dr. Moher by aUniversity of Ottawa Research Chair, and Dr. Schulz by FHI360. All researchersare independent from their relevant funding agencies.Authors contributionsLS, AS and DM have made substantial contributions to conception, designand preparing the first draft of this protocol. All authors have contributed torevising this protocol critically for important intellectual content and havegiven final approval of the version to be published. All authors read andapproved the final manuscript.AcknowledgementsThe authors would like to thank Shona Kirtley for providing details of theEQUATOR network search strategy for reporting guidelines and AndraMorrison for peer reviewing the search strategies developed for this review.Author details1Centre for Practice-Changing Research, Ottawa Hospital Research Institute,The Ottawa Hospital – General Campus, 501 Smyth Road, Box 201B, Ottawa,ON K1H 8L6, Canada. 2Independent Research and Information Consultant,Ottawa, ON, Canada. 3Centre for Statistics in Medicine, University of Oxford,Oxford, UK. 4Queen’s University, Kingston, ON, Canada. 5St. Paul’s Hospital,Vancouver, BC, Canada. 6FHI 360, Durham, NC, USA.Received: 28 February 2012 Accepted: 10 April 2012Published: 24 May 2012References1. Chan AW, Altman DG: Epidemiology and reporting of randomised trialspublished in PubMed journals. Lancet 2005, 365:1159–1162.2. Chan S, Bhandari M: The quality of reporting of orthopaedic randomizedtrials with use of a checklist for nonpharmacological therapies. J BoneJoint Surg Am 2007, 89:1970–1978.3. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology andreporting characteristics of systematic reviews. PLoS Med 2007, 4:e78.4. Smith BA, Lee HJ, Lee JH, Choi M, Jones DE, Bausell RB, Broome ME: Qualityof reporting randomized controlled trials (RCTs) in the nursing literature:application of the Consolidated Standards of Reporting Trials(CONSORT). Nurs Outlook 2008, 56:31–37.5. Yesupriya A, Evangelou E, Kavvoura F, Patsopoulos N, Clyne M, Walsh M, LinBK, Yu W, Gwinn M, Ioannidis JP, Khoury MJ: Reporting of human genomeepidemiology (HuGE) association studies: an empirical assessment. BMCMed Res Methodol 2008, 8:31.6. Zhang D, Yin P, Freemantle N, Jordan R, Zhong N, Cheng KK: Anassessment of the quality of randomised controlled trials conducted inChina. Trials 2008, 9:22.7. Moher D, Schulz KF, Simera I, Altman DG: Guidance for developers ofhealth research reporting guidelines. PLoS Med 2010, 7:e1000217.8. Moher D, Weeks L, Ocampo M, Seely D, Sampson M, Altman DG, Schulz KF,Miller D, Simera I, Grimshaw J, Hoey J: Describing reporting guidelines forhealth research: a systematic review. J Clin Epidemiol 2011, 64:718–742.9. EQUATOR Network: Library for health research reporting; http://www.equator-network.org/resource-centre/library-of-health-research-reporting/.10. Moher D: personal communication. Barriers and facilitators to CONSORTendorsement; 2011.11. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AHS, Dellinger EP,Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, ReznickRK, Taylor B, Gawande AA, Safe Surgery Saves Lives Study Group: A surgicalsafety checklist to reduce morbidity and mortality in a globalpopulation. N Engl J Med 2009, 360(5):491–499.12. Simera I, Altman DG, Moher D, Schulz KF, Hoey J: Guidelines for reportinghealth research: the EQUATOR Network’s survey of guideline authors.PLoS Med 2008, 5:e139.13. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D,Schulz KF, Simel D, Stroup DF: Improving the quality of reporting ofrandomized controlled trials, the CONSORT statement. JAMA 1996,276:637–639.14. Moher D, Schulz K, Altman D, CONSORT: The CONSORT statement: revisedrecommendations for improving the quality of reports of parallel grouprandomized trials. BMC Med Res Methodol 2001, 1:2.15. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, GøtzschePC, Lang T, CONSORT GROUP (Consolidated Standards of Reporting Trials):The revised CONSORT statement for reporting randomized trials:explanation and elaboration. Ann Intern Med 2001, 134:663–694.16. Schulz K, Altman D, Moher D, CONSORT Group: CONSORT 2010 Statement:updated guidelines for reporting parallel group randomised trials. BMJ2010, 340:c332.17. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ,Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation andelaboration: updated guidelines for reporting parallel group randomisedtrials. BMJ 2010, 340:c869.18. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Doesthe CONSORT checklist improve the quality of reports of randomisedcontrolled trials? A systematic review. Med J Aust 2006, 185(5):263–267.19. Moher D, Plint AC, Altman DG, Schulz KF, Kober T, Galloway EK, Weeks L,Dias S: Consolidated Standards of Reporting Trials (CONSORT) and the Qualityof Reporting of Randomized Controlled Trials (Protocol). Cochrane Database ofSystematic Reviews; 2010.20. Turner L, Moher D, Shamseer L, Weeks L, Peters J, Plint A, Altman DG,Schulz KF: The influence of CONSORT on the quality of reporting ofrandomized controlled trials: an updated review. Trials 2011,12(suppl 1):A47.21. Smidt N, Rutjes AWS, Van der Windt D, Ostelo R, Bossuyt PM, Reitsma JB,Bouter LM, de Vet HC: The quality of diagnostic accuracy studies sincethe STARD statement: has it improved?. Neurology 2006, 67:792–797.22. Prady SL, Richmond SJ, Morton VM, MacPherson H: A systematicevaluation of the impact of STRICTA and CONSORT recommendationson quality of reporting for acupuncture trials. PLoS One 2008, 3:e1577.23. Rajkumar SV, Harousseau J, Durie B, Anderson KC, Dimopoulos M, Kyle R,Blade J, Richardson P, Orlowski R, Siegel D, Jagannath S, Facon T, Avet-Loiseau H, Lonial S, Palumbo A, Zonder J, Ludwig H, Vesole D, Sezer O,Munshi NC, San Miguel J, International Myeloma Workshop ConsensusPanel 1: Consensus recommendations for the uniform reporting ofclinical trials: report of the International Myeloma Workshop ConsensusPanel 1. Blood 2011, 117:4691–4695.24. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C: Anevidence-based practice guideline for the peer review of electronicsearch strategies. J Clin Epidemiol 2009, 62(9):944–952.25. Reference Manager; 2008.26. Cochrane Effective Practice and Organisation of Care Review Group (EPOC):Data Collection Checklist; 2002.27. von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, VandenbrouckeJP, STROBE Initiative: Strengthening the Reporting of ObservationalStudies in Epidemiology (STROBE) statement: guidelines for reportingobservational studies. BMJ 2007, 335:806–808.28. Wells GB, Shea B, O'Connell D, Peterson J, Welch V, Losos M, Tugwell P: TheNewcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomisedstudies in meta-analyses. Liverpool, UK: University of Liverpool; 2000.29. Higgins JPT, Altman DG: Assessing risk of bias in included studies. InCochrane Handbook for Systematic Reviews of Interventions. 510th edition.Edited by Higgins JPT, Green S: The Cochrane Collaboration; 2008:187–242.30. Deeks JJ, Higgins JPT, Altman DG: Analysing data and undertaking meta-analyses. In Cochrane Handbook for Systematic Reviews of Interventions. 510thedition. Edited by Higgins JPT, Green S: The Cochrane Collaboration; 2011.31. The Cochrane Collaboration, Review Manager (RevMan). 2011; 5.1.32. Borenstein M, Hedges L, Higgins J, Rothstein H: ComprehensiveMeta-analysis. 2005.33. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group: Preferredreporting items for systematic reviews and meta-analyses: the PRISMAstatement. BMJ 2009, 339:b2535.34. Hirst A, Altman DG: Are peer reviewers encouraged to use reportingguidelines? A survey of 116 health research journals. PLoS ONE 2012, 7(4):e35621. doi:10.1371/journal.pone.0035621.doi:10.1186/2046-4053-1-24Cite this article as: Shamseer et al.: Does journal endorsement ofreporting guidelines influence the completeness of reporting of healthresearch? A systematic review protocol. Systematic Reviews 2012 1:24.Shamseer et al. Systematic Reviews 2012, 1:24 Page 9 of 9http://www.systematicreviewsjournal.com/content/1/1/24


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items