UBC Faculty Research and Publications

The challenges of including sex/gender analysis in systematic reviews: a qualitative survey Runnels, Vivien; Tudiver, Sari; Doull, Marion; Boscoe, Madeline Apr 10, 2014

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13643_2013_Article_202.pdf [ 223.2kB ]
JSON: 52383-1.0221391.json
JSON-LD: 52383-1.0221391-ld.json
RDF/XML (Pretty): 52383-1.0221391-rdf.xml
RDF/JSON: 52383-1.0221391-rdf.json
Turtle: 52383-1.0221391-turtle.txt
N-Triples: 52383-1.0221391-rdf-ntriples.txt
Original Record: 52383-1.0221391-source.json
Full Text

Full Text

RESEARCH Open AccessThe challenges of including sex/gender analysisin systematic reviews: a qualitative surveyVivien Runnels1*, Sari Tudiver2, Marion Doull3 and Madeline Boscoe4AbstractBackground: Systematic review methodology includes the rigorous collection, selection, and evaluation of data inorder to synthesize the best available evidence for health practice, health technology assessments, and healthpolicy. Despite evidence that sex and gender matter to health outcomes, data and analysis related to sex andgender are frequently absent in systematic reviews, raising concerns about the quality and applicability of reviews.Few studies have focused on challenges to implementing sex/gender analysis within systematic reviews.Methods: A multidisciplinary group of systematic reviewers, methodologists, biomedical and social scienceresearchers, health practitioners, and other health sector professionals completed an open-ended survey prior to atwo-day workshop focused on sex/gender, equity, and bias in systematic reviews. Respondents were asked toidentify challenging or ‘thorny’ issues associated with integrating sex and gender in systematic reviews and indicatehow they address these in their work. Data were analysed using interpretive description. A summary of the findingswas presented and discussed with workshop participants.Results: Respondents identified conceptual challenges, such as defining sex and gender, methodologicalchallenges in measuring and analysing sex and gender, challenges related to availability of data and data quality,and practical and policy challenges. No respondents discussed how they addressed these challenges, but allproposed ways to address sex/gender analysis in the future.Conclusions: Respondents identified a wide range of interrelated challenges to implementing sex/genderconsiderations within systematic reviews. To our knowledge, this paper is the first to identify these challenges fromthe perspectives of those conducting and using systematic reviews. A framework and methods to integrate sex/gender analysis in systematic reviews are in the early stages of development. A number of priority items andcollaborative initiatives to guide systematic reviewers in sex/gender analysis are provided, based on the surveyresults and subsequent workshop discussions. An emerging ‘community of practice’ is committed to enhancing thequality and applicability of systematic reviews by integrating considerations of sex/gender into the review process,with the goals of improving health outcomes and ensuring health equity for all persons.Keywords: Gender, Qualitative methods, Sex, Sex/gender analysis, Systematic review methodsBackgroundSound decisions about clinical interventions and aboutimplementation of health policies ideally require astrong evidence base combined with judgment abouthow the evidence could best be applied to a particularpatient and/or in a particular policy context. Systematicreviews provide a framework, methods, and tools tocollate and analyse relevant evidence from primaryresearch. They are widely used to guide clinical, policyand program decisions, conduct health technology as-sessments, and determine directions for new research[1-4]. They are also used to assess how interventionsmay contribute to health equity [5]. However, many sys-tematic reviews do not clearly address the applicabilityof the evidence [6]. Of particular interest here, most donot explicitly describe whether the evidence applies toboth women and men, nor analyse possible sex andgender-related differences and similarities, nor indicatewhen such data are missing [7,8].* Correspondence: vrunnels@uottawa.ca1Globalization and Health Equity Research Unit, Institute of PopulationHealth, University of Ottawa, 1 Stewart Street, Ottawa, ON K1N 6N5, CanadaFull list of author information is available at the end of the article© 2014 Runnels et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the CreativeCommons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andreproduction in any medium, provided the original work is properly credited. The Creative Commons Public DomainDedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article,unless otherwise stated.Runnels et al. Systematic Reviews 2014, 3:33http://www.systematicreviewsjournal.com/content/3/1/33Sex refers to the biological, genetic, and physiologicalprocesses that generally distinguish females from maleswhile gender refers to the roles, relationships, behav-iours, relative power, and other traits that societiesgenerally ascribe to women, men, and people of diversegender identities (e.g., transgender persons) [9,10]. Al-though often categorized as binary for analysis (e.g., male/female; masculine/feminine), processes of sex/gender aredynamic, multidimensional, and interactive [11-13]. Sexand gender analysis, referred to here as sex/gender ana-lysis to emphasize the interrelationships among the con-cepts, is an analytical framework that is used to explorepossible biological and social similarities and differencesbetween and among men and women, boys and girls [9].For example, in the context of health care systems, sex/gender analysis explores the interrelationships of sex andgender within or between groups in order to identify howthese may affect health experiences, access to care, andhealth outcomes.The lack of attention to sex/gender in systematic re-views is of concern for several reasons. There is a substan-tive and growing body of knowledge calling attention tosex/gender-related differences in pharmacokinetics andpharmacodynamics, genetic expression, and in prevalence,onset, and severity of diseases such as auto-immune disor-ders [14-21]. Research studies also identify sex/gender dif-ferences in subjective experiences of conditions such asdepression and chronic pain, responses to health interven-tions, the utilization of health care systems, and in manyother processes that influence health [22-25]. Failure toconsider whether such differences are clinically meaning-ful, and how processes related to sex/gender may or maynot be relevant to a particular intervention, can result ininappropriate application of findings and have harmfulimplications for clinical outcomes and/or for the successof health policies and programs [26,27].Systematic reviews can also be used to help determinewhich interventions reduce inequities in health, definedas preventable and therefore unfair differences in healthbetween groups [28-31]. Explicitly identifying to whomthe evidence does or does not apply is necessary to for-mulate social policy initiatives to reduce poverty andother inequities and to determine what interventions areappropriate with particular populations. It is widely rec-ognized that sex/gender-related processes are determi-nants of health – men, women, and persons of diversegender identities may experience disadvantages andmarginalization in different ways, both within and be-tween communities [32-35]. Further, many forms of bias,including publication bias (the tendency to publish posi-tive rather than negative research findings); and outcomereporting bias (the selective reporting of some outcomes,but not others, depending on the nature and direction ofthe results), are a major concern for systematic reviewmethods and can affect applicability and quality of stud-ies [36]. Lack of reporting about sex/gender in system-atic reviews can elevate risk of bias and limit reviewers,researchers and other users of reviews from assessing inwhat ways, and specifically for whom, interventions mayor may not be effective and may reduce or exacerbatehealth inequities.Furthermore, many research and governmental organi-zations have policies mandating that sex/gender-relatedissues be addressed in proposals for health researchfunding, clinical trial design and registration, and in pol-icy and program initiatives, citing the rationale that con-sideration of possible sex/gender-related similarities anddifferences is essential for good science, policies, andprograms [37-39]. Editors of a number of major healthjournals are beginning to call for sex/gender policies toenhance the quality and applicability of reporting of re-search evidence [40-43].Integrating sex/gender analysis into the processes andmethods of systematic reviews can enhance the applic-ability and quality of reviews, with the long term goalsof improving health outcomes and reducing health in-equities. While many review authors, editors, and groupcoordinators acknowledge there is value to consideringsex/gender, they point to challenges in implementingsuch analysis. To address these concerns, our WorkingGroup on Sex and Gender Analysis in SystematicReviewsa convened a meeting of review authors, reviewgroup coordinators and editors, methodologists, healthpractitioners, biomedical and social science researchers,consumers, and funders to discuss the factors that hin-der the integration of sex/gender analysis within system-atic reviews and to outline priorities for addressing thesechallenges. Invited participants were individuals weknew or were known to us through research, academic,and policy networks as possessing knowledge, practicalexperience, and expertise in relation to systematic re-views and/or in the application of sex/gender analysis tohealth evidence. To our knowledge, this paper is the firstto identify the challenges of integrating sex/gender ana-lysis in systematic reviews from the perspectives of thoseconducting and using systematic reviews.MethodsA brief, open-ended survey was distributed by email twoweeks prior to a two-day workshop focused on sex/gen-der, health equity, and bias in systematic reviews.b Thoseinvited were asked to briefly describe their backgrounds,how they came to be interested in systematic reviewsand quality in health research, and what outcomesthey would like to see from the meeting. The centralquestions asked were: “What do you perceive to be thechallenges of incorporating sex/gender, equity, and biasin systematic reviews? What are some of the mostRunnels et al. Systematic Reviews 2014, 3:33 Page 2 of 10http://www.systematicreviewsjournal.com/content/3/1/33perplexing, stimulating or ‘thorny’ questions/issues thathave emerged in your work with systematic reviews?and How have you attempted to address these?” [Res-pondents could select challenges and/or ‘thorny’ issuesto address the latter question]. The main goals of thesurvey were to provide organizers with a better under-standing of the interests of workshop participants in sys-tematic reviews and/or in sex/gender-related issues, toidentify a diverse range of challenges beyond what wemight have considered, and to encourage respondents tothink about challenges and possible solutions to betterinform workshop discussions.AnalysisAll responses to the questions were coded systematicallyby one analyst (VR) using qualitative data analysis soft-ware to organize the coded responses.c Units of codingwere determined as having discrete meaning as the ana-lyst interpreted them [44-46]. The data were analyzed inaccordance with qualitative description, a low-inferenceinterpretive approach to qualitative data analysis, anddesigned specifically for the purpose of description[44,47]. Each unit was first organized and then coded asa response to a specific question. Some responses wereassigned more than one code, where appropriate. Subse-quently, data were analyzed across the organized codes(i.e., in response to the questions posed) to look for pat-terns or themes in the coding. Raw data, analysis, andcoding were shared and discussed with a second analyst(ST); adjustments to coding and analysis were made byconsensus. A summary of the analyzed results werepresented to all 33 workshop participants, includinggraduate students, an invited discussant, and several re-searchers who had not been part of the survey.For reporting purposes, all possible identifiers of re-spondents were removed from the survey texts. Quota-tions are denoted by quotation marks and reportedanonymously by respondent number. Ethical commit-ments to anonymity and confidentiality, while limitingdescription of individuals, also provides the conditionsin which we can generally assume the truthfulness of re-spondents. Ethical approval was granted by the Univer-sity of Ottawa Research Ethics Board.ResultsA total of 23 of 24 individuals who accepted an initial in-vitation to attend the workshop responded to the survey.Of the 23 respondents, 3 were men and 20 were women.The estimated ages of respondents ranged from the late-20s to late-60s. Respondents came from a variety of aca-demic, clinical, health policy and other experientialbackgrounds and included university-affiliated and inde-pendent researchers, research coordinators and health-related professionals, some of whom were affiliated withnon-governmental health organizations. Systematic re-views were used by respondents as sources of evidencefor academic work, clinical practice, and/or policy devel-opment. Of the 23 respondents, 14 had experience withsystematic reviews either as members of teams who con-duct reviews or as contributors or advisors concerningreview content or methods. Most were members ofCochrane and Campbell Collaborations’ methods and re-view groups and several were affiliated with other researchorganizations conducting reviews. Nine of the 23 respon-dents indicated a primary expertise in the area of sex/gen-der analysis. The respondents’ combined experience withsystematic reviews and/or sex/gender was extensive andthey formed a multi-disciplinary group with diverse aca-demic backgrounds, approaches, and discourses.Respondents provided rich and detailed answers to thequestions, placing particular emphasis on the identifica-tion of challenges and thorny issues. No respondents of-fered examples of how they addressed these issues intheir work. All respondents identified expected out-comes from the workshop. A summary of the results isprovided in the following sections.Challenges and thorny issuesAnalysis of the data revealed a number of commonthemes among the perceived challenges to integratingconsiderations of sex/gender in systematic reviews.These have been organized into four main interrelatedcategories: 1) conceptual challenges; 2) methodologicalchallenges; 3) challenges related to availability and qual-ity of data; and 4) practical and policy challenges.Conceptual challengesClarifying the concepts of ‘sex and gender’ and puttingconcepts into practice in systematic reviews were identi-fied as significant challenges. Respondents noted the ten-dency for sex to be used as a proxy for gender and thatthe terms sex and gender were used interchangeably:“One tricky thing has been the lack of clarity aroundwhether it is gender or sex or both being reported inprimary studies and in reviews. This makes us have touse the ever-ambiguous ‘and/or’ much of the time”(Respondent #6).“Some reviewers (and other clinical researchers) maystill be unclear about the concepts of ‘sex’, ‘gender’,‘equity’, ‘race/ethnicity’, etc. and how to incorporatethese” (Respondent #21).Others pointed to the complexity of the concepts ofsex and gender, indicating that they are not single vari-ables, but interrelated biological and social processes thatin many ways have yet to be understood. RespondentsRunnels et al. Systematic Reviews 2014, 3:33 Page 3 of 10http://www.systematicreviewsjournal.com/content/3/1/33stressed the need for communication between sex/genderexperts and systematic reviewers to provide clarity andguidance on capturing the nuanced and intersecting na-ture of sex/gender particularly within the context of sys-tematic reviews.Beyond the definitional challenges, respondents calledattention to broader conceptual challenges that oftenmanifest within interdisciplinary projects. Specifically,respondents raised concerns about the construction ofknowledge and ‘what counts’ as evidence and the chal-lenges these present to incorporating sex/gender analysisinto systematic reviews. The absence of sex/gender con-siderations within systematic reviews was traced back towhat influences how knowledge is produced:“The biggest challenges are much more fundamentaland have to do with the way that we arrive atdecisions as to what is important for us to study, whyit is important for us to study, and how we determinethe way to study and ultimately produce evidence”(Respondent #3).“The emergence of an evidence-based medicineapproach has altered…our understanding of how toidentify credible evidence and how to synthesizethat evidence to inform medical practices as wellas policies and programs… This approach toevidence, knowledge and policy-making hasprofound effects on who is authorized to providecredible accounts, what counts as credible evidence,and what counts as credible synthesis andtranslation” (Respondent #16).The assumptions about ‘credible’ evidence were seento have particular implications for the inclusion of sex/gender analysis in systematic reviews because sex-disaggregated data and sex/gender differences and simi-larities are often not taken into account in the researchdesign, conduct, data analysis, and reporting of ran-domized controlled trials that form the basis of mostreviews. In studies that include both men and women,conclusions are often presented as applicable to all‘subjects’. This is, in part, related to a history of ex-cluding women in adequate numbers from many clin-ical trials and the assumption that findings fromstudies of men could be extrapolated to women [48].As respondents noted:“Over time, I realised that women were being excludedfrom the development of evidence in a variety ofways… (and this had implications for) who werescientists, who defined the questions, who were subjectsin research, what conditions were studied…”(Respondent #11).“[My] concern is a lack of examination of sex/genderand other diversity within systematic reviews. Thereare important differences but does the currentmethodology hide those differences or fail to examinethem… (Respondent #14).Dealing with diversity and heterogeneity of popula-tions was a significant methodological challenge de-scribed by respondents as detailed below.Methodological challengesMethodological issues generated the largest amount ofcommentary. Some respondents expressed concernsabout a lack of transparency in clinical trial design, out-comes, and reporting current practices which have im-plications for the quality of systematic reviews:“There is a lack of transparency about clinical trialdesign, including a priori outcomes and analyses, orpoor quality protocols. There is insufficient reporting ofmethodologic information in current trial registries…”(Respondent #21).The quality of reporting in primary studies has a cas-cading effect on reviews and on the analyses that can becompleted:“There are many perplexing issues with regards toconduct of systematic reviews. The most profound isquality of reporting. The quality of reporting directlyleads to the comprised validity of systematic reviews,as well as contributing to the assessment of clinicalheterogeneity which is a complex issue often overlookedby reviewers.” (Respondent #10).In particular, respondents noted that sex and genderand other health determinants are often neglected in thesingle studies that form the basis of reviews. They identi-fied major challenges to both primary studies and sys-tematic reviews:“…determining ways of measuring equity and bias,challenges of being able to capture complex socialprocesses in what we mean by incorporating sex/gender.” (Respondent #5).“… [the] question to ask is both related to quality ofthe research being reported and sorting out when, if, orhow sex and gender matter.” (Respondent #11).“There needs to be more work undertaken to explorehow to identify and synthesize evidence oninterventions to address structural determinants ofhealth/health equity.” (Respondent #16).Runnels et al. Systematic Reviews 2014, 3:33 Page 4 of 10http://www.systematicreviewsjournal.com/content/3/1/33An approach suggested to address these challenges isthrough use of a range of data sources and a variety ofresearch designs:“I’m wondering about the methodology and quality ofthe reviews that incorporate these different researchdesigns and how we can use all the literature, not justthe quantitative…to ensure a comprehensive overviewof the literature, including grey literature, when we aremaking context-sensitive, evidence-based decisions.”(Respondent #14).Respondents noted a lack of available tools, checklists,and/or outlines for conducting sex/gender analysis withinprimary studies and systematic reviews. Tools and check-lists are practical aids that incorporate recognition andunderstanding of what sex/gender can mean when appliedinto practice, and were seen as essential for those new tothe concepts of sex/gender. It was recognized, that “toolsare still in development and lagging behind the theoreticalarguments for inclusion of these issues” (Respondent #9).While tools are essential, caution was expressed aboutrelying too heavily on a tool to conduct complex analyses:“I struggle with the fact that I have not been able – andI am reluctant to anyway – turn sex- and gender-basedanalysis (or any other form of equity analysis) into a setchecklist of questions and techniques. To date, myexperiences have taught me that these analyses are morefluid and emergent than predictable, and depend on thehistory of research and evidence in the field, the framingof the question, the sources of data available, whethersex disaggregated data are available, how researchersoperationalize the two concepts and the context of thequestion.” (Respondent #13).Moving from the analytical framework of sex/genderanalysis towards appropriate and robust methods to ad-dress these concepts in systematic reviews raised add-itional challenges. Respondents warned about pitfallswhen developing appropriate methods to integrate sex/gender in systematic reviews. In particular, they recog-nized that subgroup analyses are an important tool forinvestigating heterogeneity and examining results formen and women separately, but such analyses raise im-portant technical and interpretive challenges:“Heterogeneous outcomes and small studies makerobust subgroup analysis difficult to complete.”(Respondent #3).“Few primary studies pre-specify subgroup analyses oracknowledge the limitations of analyses as hypothesis-generating. Over-interpretation of results (emphasis onthe subgroup analysis rather than the primaryoutcome) and subsequent claims also presentchallenges when the emphasis is on subgroup analysisrather than the primary outcome, and give rise topotential for publication bias, and spurious (not tomention some notorious…) subgroup claims in theliterature.” (Respondent #21).On the other hand, respondents considered it import-ant to address the implications of potential sex/gender-based differences and equity issues in relation to healthconditions and interventions:“Reviewers may not recognize that even smalldifferences in the magnitude of treatment effect couldtranslate into meaningful differences to consider inrisk-benefit ratios.” (Respondent #21).Respondents acknowledged that methodological chal-lenges, such as quality of data reporting and appropriatesubgroup analyses, were not unique to sex/gender con-siderations but were of general concern to systematic re-viewers. However, they identified a specific need fordialogue and guidance on the application of sex/genderanalysis to systematic review methods:“[There is a] need for appropriate guidance andtraining for both trialists and systematic reviewers andfor leadership in methods, and examples of applyingthese methods.” (Respondent #17).Challenges related to availability and quality of dataLinked to the methodological issues were challenges re-lated to the availability and quality of data needed toconduct a robust review: these challenges became par-ticularly marked when matters of sex/gender were con-sidered. Many commented on the lack of data aboutwomen, lack of sex-disaggregated data, and/or inconsist-encies in reporting such data:“[There is a] lack of sex-disaggregated data in publishedstudies. Lack of transparency in published studies, forexample, cases where some outcomes are reported forboth men and women and others are not, withoutexplanation…. [I was] working on a review thatexamined a health issue specifically among women. Itwas challenging to find data on women to conduct thereview. This stimulated questions around quality, thequestions researchers ask and the answers that resultfrom these very specific questions.” (Respondent #3).“Original studies do not include the kind ofdisaggregated data that might allow secondary sex/gender analyses to be carried out.” (Respondent #12).Runnels et al. Systematic Reviews 2014, 3:33 Page 5 of 10http://www.systematicreviewsjournal.com/content/3/1/33The absence of evidence or the use of suboptimal datawere regarded as issues of data quality that affect theconduct of systematic reviews generally. These were seento stem from “…lack of transparency, access to all thestudies, all measured outcomes in studies and to the ori-ginal data that might allow additional questions to beanswered” (Respondent #12). Respondents also pointedto the potential for serious consequences from the ab-sence, omission or other shortcomings in data gathering,analysis, and reporting, such as failure to include dataabout adverse events.“A ‘thorny’ issue that deserves special mention is howto find, adequately assess, and incorporate harms datainto intervention systematic reviews. This is aconsistent problem due to poor quality reporting and/or measurement, suppression of data or delays inpublication, short duration of follow-up, and a lack ofrepresentation of potentially vulnerable populationsfor whom the intervention is ultimately targeted.”(Respondent #21).In particular, if questions regarding sex and genderwere not asked, either when conducting single trials orsystematic reviews, then the data would not be availableto answer questions about possible differences in treat-ment effects for women or men.“I think there is an assumption by some, that if thetreatment works, why wouldn't it work for everybody?And for those convinced of the importance, many willconsider it a great effort to try to obtain thisadditional data from the authors of the originalstudies.” (Respondent #19).In essence, the non-availability and/or non-reportingof data created an iterative cycle in which questionsabout sex/gender were repeatedly left out. Respondentsaptly summarized this issue:“Questions not asked and therefore not answered.”(Respondent #3).“…the questions that are asked are often driven by thedata that are available [so that] there is someresistance to asking sex/gender/diversity questions.”(Respondent #13).These challenges underscore the importance of sys-tematic reviews in driving reporting standards for pri-mary studies. Emerging guidance on the use of sex/gender analysis in systematic reviews encourages authorsto report what is known and not known about effective-ness of the intervention reviewed [49].Practical and policy challengesIn addition to the difficulties of obtaining sex-disaggregateddata, and a lack of tools, respondents identified otherpractical challenges to integrating sex/gender in system-atic reviews in health. These included an absence of‘model’ reviews to illustrate how sex/gender analysis couldbe carried out:“There are still relatively few examples of solidCochrane reviews that incorporate a consistentapproach to sex and gender analysis and equityanalysis.” (Respondent #20).Other practical challenges dealt with the realities ofworking in a research intensive environment with com-peting demands and expanding workloads. Respondentscited experiences of resistance or ‘push back’ to incorp-orating considerations of sex, gender, and equity in sys-tematic reviews “…from authors and review groups whofeel overwhelmed with the workload already demandedof them” (Respondent #9). Respondents also discussedrealities of publishing, noting “[there is] difficulty findingjournals that think this sort of research has merit” (Re-spondent #7). In order to address these challenges, re-spondents saw the need for researchers, reviewers andeditors to be aware of the substantive and growing bodyof health evidence addressing sex/gender similarities anddifferences in relation to health outcomes. A criticalmass of evidence was considered integral to support themerit of such studies.Respondents also identified several policy challengesthat were system level and generally fell outside thescope of systematic review practice. Relating to earlierchallenges about the availability of data, respondentsdescribed a need for policies that would mandate trans-parency in research and data collection, calling for in-creased transparency and more complete reporting ofmethodological information in current trial registries.These issues have important implications for the qualityof systematic reviews.“[There is] a lack of binding guidance on thepart of regulatory agencies and funders of clinicaltrials to collect the data in a pre-specifiedmanner, and to make the data publicly available.”(Respondent #21).The types of decisions required to address these chal-lenges are typically made and funded by higher levels ofgovernment or quasi-governmental bodies that imple-ment government directions into policy. However, sys-tematic reviewers working at the ‘coalface’ of reportedstudy data have an important role to play in advocatingfor such policy changes.Runnels et al. Systematic Reviews 2014, 3:33 Page 6 of 10http://www.systematicreviewsjournal.com/content/3/1/33Proposed outcomesWhile respondents described the challenges and thornyissues in detail, no one answered the question, “Howhave you attempted to address these?” However, respon-dents did propose a range of outcomes they hopedwould emerge from the workshop. These proposed out-comes indicate how some of the identified challengescould be approached and eventually addressed. Proposedoutcomes included developing methods guidelines and/or a series of methods papers to provide a frameworkand practical ways to integrate sex/gender analysis insystematic reviews and in peer review; refining currenttools and checklists for the design and appraisal of re-views and/or developing new tools; and establishingcross-disciplinary collaborations through a ‘communityof practice’. Respondents sought ways to determine the“threshold of important differences” of sex/gender (Re-spondent #13), the influence of sex/gender “on the sum-mary effects in a systematic review” (Respondent #1),and “the key ideas to help increase the sex/gender rele-vance of reviews…the minimum to consider to maximizethe impact” (Respondent #8). Through joint efforts, col-laborators could share resources, conduct research, in-cluding model reviews, and provide educational fora,training and outreach to biomedical and social scientists,practitioners, peer reviewers, journal editors, fundersand policy makers interested in the quality and applicabil-ity of health evidence. Collaboration might also result in“a strategy to encourage funding agencies and regulatoryagencies to require the collection of key data related to is-sues of sex/gender, equity, and bias in publicly funded tri-als, full disclosure of study results, and the publication ofprotocols” (Respondent #21). Many of these suggestionslaid the basis for initiatives discussed below.DiscussionSystematic review methodology provides a frameworkand structure for the collection, analysis, synthesis, andreporting of data, and notably in the case of CochraneReviews, strict protocols and methodological approachesthat are regularly reviewed and updated [1]. However,despite considerable research demonstrating that sex-and gender-related processes often matter to health out-comes and an increasing number of institutional policiessupporting sex/gender analysis in health research andreporting, these issues remain largely unknown or disre-garded. This is the case for many areas of medical andclinical research, health technology assessment, and inconducting systematic reviews [7,50]. Asking a multidis-ciplinary group of individuals who conduct and/or usesystematic reviews to identify what they considered tobe major challenges to addressing sex/gender analysis inreviews proved invaluable to scope out key conceptual,methodological, practical, and policy issues.In particular, respondents recognized that integratingsex/gender analysis in systematic reviews is an importantstep forward to answer the question “to whom does theevidence apply and to whom might it not apply?” inorder to make sound clinical and policy decisions. Theysought greater clarity in defining the concepts of sex andgender and guidance in operationalizing these conceptsin the context of systematic reviews. Requests for clarityand guidance were not surprising given that theseconcepts have been re-defined since the 1970s by AnnOakley, Nancy Krieger and other researchers and theore-ticians, to transcend narrow assumptions of biologicaldeterminism in relation to women’s health, and thatinternational research bodies, health organizations, andacademic researchers continue to develop analyticframeworks to address sex/gender in more nuancedways [12,51-53]. As some respondents suggested, this in-volves seeing sex and gender as more than discrete vari-ables or part of a checklist of items to be ‘controlled for’,but as dynamic and entangled biological and social pro-cesses that shape human health and cross-cut otherhealth determinants.It was also clear from responses to the survey ques-tions that respondents, a majority of whom had expert-ise in systematic review methodology, recognized themany limitations to the quality of evidence in primarystudies, systematic reviews, and meta-analyses. Theseperspectives were evident in comments about poor qual-ity protocols; the lack of transparency in clinical trialdesign; inadequate reporting of outcomes, including ad-verse events; in the challenges of conducting sub-groupanalyses; and in addressing the many forms of methodsbias in appraisal of evidence. As Ioannidis and othershave shown, seemingly rigorous methods have consider-able strengths, “but can also lead to wrong or misleadinganswers” [54, p. 169]. For example, the outcomes ofmeta-analyses on the same topic may differ widely dueto many factors, including that those conducting/spon-soring the meta-analyses may choose different datasources, search strategies, inclusion/exclusion criteria foreligible populations, and have different, sometimes con-flicting interests that may influence how results are pre-sented and interpreted [54].Systematic reviewers who choose to include consider-ations of sex/gender and equity in their protocols are oftenlimited by the absence of relevant data in primary researchstudies. Decisions taken early in the research process haveimplications for the subsequent synthesis of evidence. Assome respondents noted, asking questions about thepotential relevance of sex/gender-related processes to theresearch question or intervention at the protocol stage ofsystematic review design can help, for example, to clarifybaseline from effect differences and pre-empt other gapsand omissions with regard to sex/gender.Runnels et al. Systematic Reviews 2014, 3:33 Page 7 of 10http://www.systematicreviewsjournal.com/content/3/1/33Importantly, there is an increased interest within theCochrane Collaboration and within the larger systematicreview community as to how knowledge is produced, ap-praised and translated into practice, and in the theoryand logic behind interventions [36,55-59]. Several groupsare engaged in work to increase the quality and applic-ability of systematic reviews and in particular, to addressissues of health equity [6,60]. Current guidelines, includ-ing the Cochrane and Campbell Health Equity checklist[61], and the Equity Extension of PRISMA (PreferredReporting Items for Systematic Reviews and Meta-Analyses) [62], are examples of tools beginning to in-corporate consideration of sex/gender. There is alsoincreased interest in addressing context in reviews, forexample, by incorporating qualitative evidence aboutwho may or may not be affected by particular health in-terventions, in what ways, and why, including relevantcultural and historical conditions [63,64]. Attention tocontext can broaden the scope and methods of reviewsto better integrate sex/gender analysis, and consider-ations of equity.ConclusionsThe results of this survey identified challenges and‘thorny’ issues associated with sex/gender in systematicreviews and emphasized that methods to integrate sex/gender analysis in reviews are still in the early stages ofdevelopment. To begin to address these challenges,workshop participants suggested a number of priorityitems and collaborative initiatives, including the develop-ment of resources, such as ‘briefing notes’ and methodspapers, to guide reviewers in the design and appraisal ofsystematic reviews. As part of such guidance, reviewersare encouraged, at a minimum, to disaggregate data bysex; conduct subgroup analyses, when possible, or indi-cate why analyses could not be performed; and addressthe question: To whom does this evidence apply?Reporting what is known and not known about sex/gen-der transparently highlights gaps in available data, con-tributes to future research questions, and reinforcesawareness about the need for improved reporting prac-tices about sex/gender in primary studies.The many challenges identified in the survey may alsohelp to explain why existing policies that encourage and/or mandate the integration of sex/gender analysis inhealth research are not consistently implemented. Suchpolicies and guidelines are necessary to raise awarenessof sex/gender considerations, redress past exclusions ofwomen, minorities and others from clinical trials andimprove the quality of health evidence: they require pol-itical will, direction and monitoring to ensure implemen-tation. Based on responses to the survey and subsequentworkshop discussions about the conceptual, methodo-logical, practical, and policy challenges of integratingsex/gender analysis in systematic reviews, we suggestthat the implementation of existing sex/gender policiesand guidelines should be supported and reinforcedthrough opportunities for training, collaborative pro-jects, and cross-disciplinary dialogue among a range ofstakeholders, including trial sponsors, researchers, sys-tematic reviewers, policy makers, journal editors, re-search ethics boards and peer review bodies.Spurred by the challenges and priorities identified atthe workshop, a community of practice has emergedamong those with expertise and interest in sex/genderanalysis, health equity, and systematic review method-ology to integrate sex/gender analysis into the practiceof systematic reviews. Participants are working togetherto enhance the quality and applicability of systematic re-views, with the goals of improving health outcomes andhealth equity for all persons.EndnotesaSee: http://equity.cochrane.org/sex-and-gender-analysis.b“Combining Forces to Improve Systematic Reviews:Gender, Equity and Bias”, funded by the Canadian Insti-tutes of Health Research (Grant#109018).cNVivo qualitative data analysis software Version 82008. Doncaster, Australia: QSR International Pty Ltd.[http://www.qsrinternational.com].Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsVR conceived and developed the survey, analyzed the data, and drafted andrevised the article. ST conceived and developed the survey, analyzed thedata, and drafted and revised the article. MD conceived and developed thesurvey, and drafted and revised the article. MB conceived and developed thesurvey, and helped to revise the article. All authors read and approved thefinal manuscript.AcknowledgementsCanadian Institutes of Health Research (Grant#109018). Survey Respondentsand Participants of the meeting “Combining Forces to Improve SystematicReviews: Gender, Equity and Bias”, May 2–3, 2011 University of Ottawa,Ottawa, Ontario. The authors would also like to thank the Handling Editorand two reviewers of Systematic Reviews for helpful comments on an earlierdraft of this paper.Author details1Globalization and Health Equity Research Unit, Institute of PopulationHealth, University of Ottawa, 1 Stewart Street, Ottawa, ON K1N 6N5, Canada.2Gender and Health Consultant/Researcher, 161 Northwestern Ave, Ottawa,ON K1Y 0M1, Canada. 3School of Population and Public Health, Faculty ofMedicine, University of British Columbia, 414-2206 East Mall, Vancouver, BCV6T 1Z3, Canada. 4Reach Community Health Centre, 1145 Commercial Drive,Vancouver, BC V5L 3X3, Canada.Received: 19 November 2013 Accepted: 31 March 2014Published: 10 April 2014References1. Cochrane Handbook for Systematic Reviews of Interventions Version5.1.0 [updated March 2011]. In The Cochrane Collaboration. Edited byHiggins JPT, Green S. 2011 [www.cochrane-handbook.org]Runnels et al. Systematic Reviews 2014, 3:33 Page 8 of 10http://www.systematicreviewsjournal.com/content/3/1/332. The Cochrane Library Health Technology Assessment Database.[http://onlinelibrary.wiley.com/o/cochrane/cochrane_clhta_articles_fs.html]3. Lavis J, Davies H, Oxman A, Denis J, Golden-Biddle K, Ferlie E: Towardssystematic reviews that inform health care management and policy-making. J Health Serv Res Policy 2005, 10(Suppl 1):35–48.4. Zief S, Agodini R: Supporting Policy and Program Decisions: Recommendationsfor Conducting High Quality Systematic Evidence Reviews. [www.mathematica-mpr.com/publications/…/systematic_reviews_ib.pdf]5. Welch V, Tugwell P, Petticrew M, de Montigny J, Ueffing E, Kristjansson B,McGowan J, Benkhalti Jandu M, Wells GA, Brand K, Smylie J: How effects onhealth equity are assessed in systematic reviews of interventions.Cochrane Database Syst Rev 2010, 12:MR000028.6. Cochrane Applicability and Recommendations Methods Group. [http://armg.cochrane.org/resources-review-authors]7. Doull M, Runnels V, Tudiver S, Boscoe M: Appraising the evidence:applying sex- and gender-based analysis (SGBA) to Cochrane systematicreviews on cardiovascular diseases. J Women's Health 2010, 19(5):997–1003[http://www.liebertonline.com/doi/abs/10.1089/jwh.2009.1626]8. Johnson SM, Karvonen CA, Phelps CL, Nader S, Sanborn BM: Assessment ofanalysis by gender in the Cochrane reviews as related to treatment ofcardiovascular disease. J Women's Health 2003, 12:449–457.9. Health Canada: Federal Health Portfolio Sex and Gender-Based Analysis Policy.[http://www.hc-sc.gc.ca/hl-vs/pubs/women-femmes/sgba-policy-politique-ags-eng.php]10. Johnson JL, Greaves L, Repta R: Better Science with Sex and Gender – A Primerfor Health Research. Vancouver: Women’s Health Research Network; 2007.11. Johnson JL, Repta R: Sex and gender: Beyond the binaries. In Designingand Conducting Gender, Sex, and Health Research. Edited by Oliffe JL,Greaves L. Thousand Oaks, CA: Sage Publications; 2012:17–37.12. Springer KW, Stellman JM, Jordan-Young RM: Beyond a catalogue ofdifferences: a theoretical frame and good practice guidelines forresearching sex/gender in human health. Soc Sci Med 2011, 74(11):1–8.13. Tudiver S, Boscoe M, Runnels VE, Doull M: Challenging “dis-ease”: sex,gender and systematic reviews in health. In What a Difference Sex andGender Make: A Gender, Sex and Health Research Casebook. Edited by Coen S,Banister E. Ottawa: Canadian Institutes of Health Research; 2012:25–33[http://www.cihr-irsc.gc.ca/e/44734.html#a04]14. Soldin OP, Chung SH, Mattison DR: Sex differences in drug disposition.J Biomed Biotechnol 2011, 2011:187103.15. Anderson GD: Sex and racial differences in pharmacological response:where is the evidence? Pharmacogenetics, pharmacokinetics, andpharmacodynamics. J Women's Health 2005, 14(1):1.16. Ober C, Loisel DA, Gilad Y: Sex-specific genetic architecture of humandisease. Nat Rev Genet 2008, 9:911–922.17. Makkar RR, Fromm BS, Steinman RT, Meissner MD, Lehmann MH: Femalegender as a risk factor for torsades de pointes associated withcardiovascular drugs. J Am Med Assoc 1993, 270:2590–2597.18. Wolbrette D: Gender differences in the proarrhythmic potential of QT-prolonging drugs. Curr Womens Health Rep 2002, 2:105–109.19. Lu J, Wallace DJ, Ishimori ML, Scofield RH, Weisman MH: Review: malesystemic lupus erythematosus: a review of sex disparities in this disease.Lupus 2010, 19(2):119–129.20. Srikanth VK, Fryer JL, Zhai G, Winzenberg TM, Hosmer D, Jones G: A meta-analysis of sex differences prevalence, incidence and severity ofosteoarthritis. Osteoarthritis Cartilage 2005, 13:769–781.21. Van Vollenhoven RF: Sex differences in rheumatoid arthritis: more thanmeets the eye. BMC Med 2009, 7:12.22. Yonkers KA, Brawman-Mintzer O: The pharmacologic treatment ofdepression: is gender a critical factor? J Clin Psychiatry 2002, 63(7):610–615.23. Borkhoff C, Hawker GA, Wright JG: Patient gender affects the referral andrecommendation for total joint arthroplasty. Clin Orthop Relat Res 2011,469:1829–1837.24. Racine M, Tousignant-Laflamme Y, Kloda LA, Dion D, Dupuis G, ChoinièreMA: A systematic literature review of 10 years of research on sex/genderand pain perception – part 2: do biopsychosocial factors alter painsensitivity differently in women and men? Pain 2012, 153:619–635.25. Hurley RW, Adams MCB: Sex, gender, and pain: an overview of a complexfield. Anesth Analg 2008, 107:309–317.26. US General Accounting Office: Drug safety: most drugs withdrawn inrecent years had greater health risks for women. [http://www.gao.gov/products/GAO-01-286R]27. DunnGalvin A, Hourihane JO’B, Frewer L, Knibb RC, Oude Elberink JNG,Klinge I: Incorporating a gender dimension in food allergy research: areview. Allergy 2006, 61:1336–1343.28. Whitehead M: The concepts and principles of equity and health. HealthPromot Int 1991, 6(3):217–228.29. Braveman P, Gruskin S: Defining equity in health. J Epidemiol CommunityHealth 2003, 57:254–258.30. Welch V, Petticrew M, Ueffing E, Jandu MB, Brand K, Dhaliwal B, KristjanssonE, Smylie J, Wells GA, Tugwell P: Does consideration and assessment ofeffects on health equity affect the conclusions of systematic reviews? Amethodology study. PLoS One 2012, 7(3):e31360.31. Bambra C, Gibson M, Sowden A, Wright K, Whitehead M, Petticrew M:Tackling the wider social determinants of health and health inequalities:evidence from systematic reviews. J Epidemiol Community Health 2010,64:284–291.32. Phillips SP: Defining and measuring gender: a social determinant ofhealth whose time has come. Int J Equity Health 2005, 4(11):4–15.33. Public Health Agency of Canada: The Chief Public Health Officer’s Report onthe State of Public Health in Canada, 2012: Influencing Health – TheImportance of Sex and Gender. [http://publichealth.gc.ca/CPHOreport]34. World Health Organization: Closing the Gap in a Generation: Health Equitythrough Action on the Social Determinants of Health. Commission on SocialDeterminants of Health Final Report. [http://www.who.int/social_determinants/thecommission/finalreport/en/index.html]35. Östlin P, Schrecker T, Sadana R, Bonnefoy J, Gilson L, Hertzman C, Kelly MP,Kjellstrom T, Labonté R, Lundberg O, Muntaner C, Popay J, Sen G, Vaghri Z:Priorities for research on equity and health: towards an equity-focusedhealth research agenda. PLoS Med 2012, 8(11):e1001115.36. Cochrane Collaboration Bias Methods Group: Assessing risk of bias inincluded studies. [http://bmg.cochrane.org/assessing-risk-bias-included-studies]37. Health Canada: Guidance Document: – Considerations for Inclusion of Womenin Clinical Trials and Analysis of Sex Differences. [http://www.hc-sc.gc.ca/dhp-mps/brgtherap/applic-demande/guides/iwct_ifec-eng.php#a1]38. Canadian Institutes of Health Research: Gender, Sex and Health Research Guide:A Tool for CIHR Applicants. [http://www.cihr-irsc.gc.ca/e/32019.html]39. U.S. Department of Health and Human Services: National Institutes of Health:NIH Policy and Guidelines on The Inclusion of Women and Minorities asSubjects in Clinical Research - – Amended, October, 2001. [http://grants.nih.gov/grants/funding/women_min/guidelines_amended_10_2001.htm]40. European Association of Science Editors: Gender Policy Committee. [http://www.ease.org.uk/about-us/organisation-and-administration/gender-policy-committee]41. Heidari S, Eckert MJ, Kippax S, Karim QA, Sow PS, Wainberg MA: Time forgender mainstreaming in editorial policies. J Int AIDS Soc 2011, 14:11.42. Johnson JL, Beaudet A: Sex and gender reporting in health research: whyCanada should be a leader. Can J Public Health 2013, 104(1):e80–e81.43. Clifford TJ: Gender issues: do as I say, not as I do? Int J Technol AssessHealth Care 2011, 27(3):191–192.44. Sandelowski M: Focus on research methods: whatever happened toqualitative description? Res Nurs Health 2000, 23:334–340.45. Johnson B, Christensen L: Educational Research: Quantitative, Qualitative andMixed Approaches. 3rd edition. Thousand Oaks, CA: Sage Publications; 2008.46. Lindlof TR, Taylor BC: Qualitative Communication Research Methods.Thousand Oaks, CA: Sage Publications; 2002.47. Neergard MA, Olesen F, Andersen RS, Sondergaard J: Qualitativedescription – the poor cousin of health research? BMC Med Res Methodol2009, 9:52.48. Merkatz RB: Inclusion of women in clinical trials: a historical overview ofscientific, ethical, and legal issues. J Obstet Gynecol Neonatal Nurs 1998,27(1):78–84.49. Doull M, Runnels V, Tudiver S, Boscoe M: Sex and gender in systematicreviews: Planning tool. [http://equity.cochrane.org/sites/equity.cochrane.org/files/uploads/SRTool_PlanningVersionSHORTFINAL.pdf]50. Panteli D, Zentner A, Storz-Pfennig P, Busse R: Gender in health technologyassessment: pilot study on agency approaches. Int J Technol Assess HealthCare 2011, 27(3):224–229.51. Krieger N: Genders, sexes, and health: what are the connections – andwhy does it matter? Int J Epidemiol 2003, 32:652–657.52. Oakley A: Experiments in Knowing: Gender and Method in the Social Sciences.New York: The New Press; 2000.Runnels et al. Systematic Reviews 2014, 3:33 Page 9 of 10http://www.systematicreviewsjournal.com/content/3/1/3353. Coen S, Banister E (Eds): What a Difference Sex and Gender Make: A Gender,Sex and Health Research Casebook. [http://www.cihr-irsc.gc.ca/e/44082.html]54. Ioannidis JPA: Meta-research: the art of getting it wrong. Res SynthMethods 2010, 1:169–184.55. Snilstveit B: Systematic reviews: from ‘bare bones’ reviews to policyrelevance. J Dev Effectiveness 2012, 4(3):388–408.56. Greenhalgh T: Why do we always end up here? Evidence-based medicine’sconceptual cul-de-sacs and some off-road alternative routes. J Prim HealthCare 2011, 4(2):92–97.57. Welch V, Pettigrew M, O’Neill J, Waters E, Armstrong R, Bhutta ZA, Francis D,Koehlmoos TP, Kristjansson E, Pantoja T, Tugwell P: Health equity: evidencesynthesis and knowledge translation methods. Syst Rev 2013, 2:43.58. Anderson LM, Petticrew M, Rehfuess E, Armstrong R, Ueffing E, Baker P,Francis D, Tugwell P: Using logic models to capture complexity insystematic reviews. Res Synth Methods 2011, 2(1):33–44.59. Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, Waters E,Bonnefoy J, Morgan A, Doohan E, Kelly MP: Assessing equity in systematicreviews: realising the recommendations of the Commission on SocialDeterminants of Health. BMJ 2010, 341:c4739.60. Campbell and Cochrane Equity Methods Group: Campbell and CochraneEquity Methods Group. [http://equity.cochrane.org/]61. Ueffing E, Tugwell P, Welch V, Petticrew M: Kristjansson E for the Campbelland Cochrane Equity Methods Group. Equity Checklist for Systematic ReviewAuthors. Version 2012-10-04. [http://equity.cochrane.org/sites/equity.cochrane.org/files/uploads/EquityChecklist2012.pdf]62. Welch V, Petticrew M, Tugwell P, Moher D, O’Neill J, Waters E, White H, thePRISMA-Equity Bellagio Group: PRISMA-equity 2012 extension: reportingguidelines for systematic reviews with a focus on health equity. PLoSMed 2012, 9(10):e1001333.63. Noyes J, Popay J, Pearson A, Hannes K, Booth A: Qualitative Research andCochrane Reviews. In Cochrane Handbook for Systematic Reviews ofInterventions: Cochrane Book Series. Edited by Higgins JP, Green S.Chichester, UK: John Wiley & Sons; 2008.64. Edwards N, Di Ruggiero E: Exploring which context matters in the studyof health inequities and their mitigation. Scand J Public Health 2011,39(Suppl 6):43–49.doi:10.1186/2046-4053-3-33Cite this article as: Runnels et al.: The challenges of including sex/gender analysis in systematic reviews: a qualitative survey. SystematicReviews 2014 3:33.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitRunnels et al. Systematic Reviews 2014, 3:33 Page 10 of 10http://www.systematicreviewsjournal.com/content/3/1/33


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items