UBC Faculty Research and Publications

Towards the use of mixed methods inquiry as best practice in health outcomes research Regnault, Antoine; Willgoss, Tom; Barbic, Skye Apr 11, 2018

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-41687_2018_Article_43.pdf [ 468.1kB ]
JSON: 52383-1.0365574.json
JSON-LD: 52383-1.0365574-ld.json
RDF/XML (Pretty): 52383-1.0365574-rdf.xml
RDF/JSON: 52383-1.0365574-rdf.json
Turtle: 52383-1.0365574-turtle.txt
N-Triples: 52383-1.0365574-rdf-ntriples.txt
Original Record: 52383-1.0365574-source.json
Full Text

Full Text

COMMENTARY Open AccessTowards the use of mixed methods inquiryas best practice in health outcomesresearchAntoine Regnault1* , Tom Willgoss2, Skye Barbic3 and On behalf of the International Society for Quality of LifeResearch (ISOQOL) Mixed Methods Special Interest Group (SIG)AbstractMixed methods research (MMR) has found an increased interest in the field of health outcomes research. Considerationfor both qualitative and quantitative perspectives has become key to contextualising patient experiences in a clinicallymeaningful measurement framework. The purpose of this paper is to outline a process for incorporating MMR in healthoutcomes research to guide stakeholders in their understanding of the essence of mixed methods inquiry. In addition,this paper will outline the benefits and challenges of MMR and describe the types of support needed for designingand conducting robust MMR measurement studies. MMR involves the application of a well-defined and pre-specifiedresearch design that articulates purposely and prospectively, qualitative and quantitative components to generate anintegrated set of evidence addressing a single research question. Various methodological design options are possibledepending on the research question. MMR designs allow a research question to be studied thoroughly from differentperspectives. When applied, it allows the strengths of one approach to complement the restrictions of another. Amongother applications, MMR can be used to enhance the creation of conceptual models and development of newinstruments, to interpret the meaningfulness of outcomes in a clinical study from the patient perspective, and informhealth care policy. Robust MMR requires research teams with experience in both qualitative and quantitative research.Moreover, a thorough understanding of the underlying principles of MMR is recommended at the point of studyconception all the way through to implementation and knowledge dissemination. The framework outlined in thispaper is designed to encourage health outcomes researchers to apply MMR to their research and to facilitateinnovative, patient-centred methodological solutions to address the complex challenges of the field.Background“We need a moral and methodological community thathonors and celebrates paradigm and methodologicaldiversity.” Denzin, [1]; (pp.425).Mixed methods research (MMR) has been establishedfor more than 50 years as a methodological approach inthe social and behavioural sciences and is now wellaccepted and commonly used in health sciences [2–4]. Inline with the call to “measure what matters” to patients,patient reported outcomes are increasingly being used inclinical care and research. However, a recent review ofstudies documenting the development of patient-reportedoutcomes (PRO) measures highlights that only 11% ofPROs have been developed by actually asking patientswhich outcomes are important to them [5]. This high-lights a clear application for MMR to combining qualita-tive and quantitative methods in health research to ensurea focus on patient-identified priorities, scientific rigour,and improved patient outcomes.In 2012, a special section of Quality of Life Research wasdedicated to applied MMR [6]. In this issue, MMR was de-scribed as an approach to inform the content validity ofPROs within the early development phase. The Food andDrug Administration (FDA) also outlined a clear role forMMR in their roadmap to patient focused measurement[7]. Shortly after, a Special Interest Group (SIG) was createdwithin the International Society for Quality of Life Research(ISOQOL) to promote the use of MMR and encouragehealth outcomes researchers to embrace the MMR* Correspondence: antoine.regnault@modusoutcomes.com1Modus Outcomes, 61, Cours de la Liberté, 69003 Lyon, FranceFull list of author information is available at the end of the article   Journal of Patient-Reported Outcomes© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made.Regnault et al. Journal of Patient-Reported Outcomes  (2018) 2:19 https://doi.org/10.1186/s41687-018-0043-8paradigm. As a SIG, we believe that there is a need forguiding principles for researchers who wish to undertakeMMR. With this position paper, we aim to provide a frame-work for MMR in health outcomes by outlining the charac-teristics of this methodology, what can be expected andwhere caution should be exercised.Defining features of mixed methods researchThe application of the MMR paradigm in the health out-comes field can be rooted in the widely accepted definitionby Tashakkori and Creswell: “Mixed Methods Research is aresearch in which the investigator collects and analysesdata, integrates the findings, and draws inferences usingboth qualitative and quantitative approaches or methods ina single study or program of inquiry” [8] (pp.2).In the spirit of pragmatism which underlies this meth-odology, we assert that utilizing MMR in the health out-comes field should not be limited to the application of aclosed list of possible methodological options, butshould be viewed as a framework characterized by threekey defining features described in Table 1.In this framework, not only should both qualitative andquantitative strands be used by the researcher, but theyshould be complemented in a relevant research designthat is set a priori. They may address distinct, specific re-search questions but they contribute to the same overallend purpose of the MMR. Once the overall purpose, andthe specific research questions of the qualitative and quan-titative strands, are well-defined and procedures are out-lined, a clear plan for interaction between the qualitativeand quantitative research components is needed. Thispoint relates to the notion of meta-inference that requiresqualitative and quantitative evidence not be considered in-dependently, but interpreted together as a single body ofevidence [3]. Importantly, the importance of the specifica-tion relates to the research design, and especially the ar-ticulation of the qualitative and quantitative strands, but itobviously does not necessary apply within the researchstrands, as, in many instances, in particular for qualitativeresearch, a full prespecification may not be appropriate,the research being of exploratory nature.The characterization of MMR is driven neither by the datacollection process, nor by the analysis technique. In MMR, itis not necessary that the qualitative and quantitative streamsinvolve data collected with the same respondents. A well-designed MMR study may combine qualitative and quantita-tive data from different samples of individuals to address asingle research question, combining rigorous qualitative andquantitative evidence. Conversely, the collection of qualita-tive and quantitative data for the same individuals does notnecessarily allow for a proper MMR solution as it may bedone to address different research questions or without con-sidering both data sets in an integrated approach.Many options are available to the health outcomesresearcher looking to utilize MMR, depending on the re-search question and design. We assert that MMR shouldnot be restricted to any specific research design or meth-odology, but rather the design that is best suited to answerthe research question posed. The articulation of thequalitative and quantitative elements can be performed invarious designs that are well described in the methodo-logical literature (e.g. convergent, or parallel or concurrentdesigns; sequential designs; embedded designs) [4]. Thechoice of the appropriate design and analysis technique(qualitative and quantitative) remains the responsibility ofthe researcher who should be guided by the principlesoutlined above.Benefits and challenges of mixed methods researchBenefitsMMR allows a research question to be studied from dif-ferent perspectives. For example, one can combine therich, subjective insights on complex realities from quali-tative inquiry, with the standardized, generalizable datagenerated through quantitative research. When applied,MMR allows respective strengths and weaknesses ofeach approach to complement each other.Since its conception in 2015, the Mixed Methods SIGof ISOQOL has identified and discussed many differentapplications of MMR in the health outcomes field: ex-ploration of patient experiences to support the develop-ment of conceptual models with group concept mapping[9–11]; development of new clinical outcome assessmentinstruments with integrated qualitative and early quanti-tative analyses with Rasch model [12–15]; quantitatiza-tion of qualitative data to support conceptual saturationanalyses [16, 17]; use of qualitative information to sup-port the interpretation of quantitative patient-reportedoutcomes results [18, 19], to name but a few.These examples show that MMR can help us to addresscommon questions in the health outcomes field. This istypified by the inductive and iterative process characteristicof the development of new PROs. It also allows for flexibil-ity and the ability to make the most of small samples.MMR enables a pragmatic path forward to conduct healthoutcomes measurement research in rare disease popula-tions [20] or populations that are often difficult to recruitTable 1 Defining features of the mixed methods researchframework1. A specific research question is to be addressed using quantitativeand qualitative components (data and/or methods)2. The quantitative and qualitative components are articulatedpurposely and prospectively in a well-defined, pre-specifiedresearch design3. The response to the research questions is supported by anintegrated set of evidence generated from both the qualitativeand quantitative component of the research (meta-inference)Regnault et al. Journal of Patient-Reported Outcomes  (2018) 2:19 Page 2 of 4for research purposes (e.g., paediatrics, acute mental health,palliative). In a clinical research context, the versatility ofMMR makes it a method of choice for hypothesis gener-ation on PRO endpoints, especially in phase II trials.Finally, a critical strength of MMR approaches is thatthey typically capitalize on data reflecting individual livedexperiences (in the qualitative strand). This ensures that theresults are considered from the patient-perspective. Incorp-orating the patient voice in MMR helps ensure that the re-search is focused on the needs and priorities of patients.Moreover, MMR can facilitate the involvement of other keystakeholders, such as partners, family members, and/orother knowledge users, in the process of developing the re-search question(s) and outlining the research designs. Inthis context, it appears clearly that MMR is a strong optionto leverage effective patient engagement and support on-going research focused on patient-identified priorities andthe improvement of patient outcomes [21].ChallengesDespite some clear benefits, the application of MMR inthe health outcomes research does not come without chal-lenges. One major hurdle is that MMR is demanding interms of methodological skillsets. MMR requires a teamof researchers who are experienced in both qualitative andquantitative research, and in MMR designs. Indeed, aswith traditional qualitative or quantitative methodologies,best practices should be applied rigorously across the mul-tiple methods, but also in the way the quantitative andqualitative strands are articulated. A particularly criticalissue in this context is that of meta-inference, in whichthe qualitative and quantitative strands connect. Meta-inference should be carefully specified, and researchersshould be aware of the challenges of interpreting conflict-ing results.The application of MMR can also raise practical con-siderations, particularly as the integration of both quali-tative and quantitative data can require additionalresources and time. However, it should be noted thatthis additional burden can often be offset against the po-tential benefits of MMR, particularly where multiple in-sights support the investigation of complex researchquestions or small populations.Finally, we acknowledge that some theoretical debatestill exists on how - or even whether - quantitative andqualitative paradigms can be mixed, a debate typified bythe ‘paradigm wars’ of the second half of the twentiethcentury [1]. Such challenges stem from differences inthe underlying ontological and epistemological positionsof positivism (that a single objective reality exists) andconstructivism (that reality is a subjective construct andtherefore multiple realities exist). Even though the MMRparadigm goes beyond simply mixing the quantitativeand qualitative paradigms and builds a third path, somepurists continue to question this third paradigm, consid-ering the very nature of the qualitative and quantitativeparadigms irreconcilable. However, as Maxwell and Mit-tapalli argue [22], there is an alternative position to posi-tivism and constructivism – critical realism. Criticalrealists deny that we have any objective or certain know-ledge of the world, and accept the possibility of alterna-tive valid accounts of any phenomenon. They argue thatall theories about the world are grounded in a particularperspective and worldview, and all knowledge is partial,incomplete, and fallible. As such, critical realism pro-vides a philosophical stance that is compatible withMMR in that it acknowledges the methodological char-acteristics of both qualitative and quantitative research,and can facilitate communication and cooperation be-tween the two. Against the background of this ongoingdebate, it is clear from the growing literature that the ac-ceptance of MMR is increasing as the health outcomesresearch community continues to promote and celebratemethodological diversity.Conclusions and recommended readingTwo conditions appear critical for the continued develop-ment of credible and robust MMR. First, health outcomesresearchers have the potential to learn about the differentMMR methodologies and outline how MMR can be usedto more thoroughly answer health outcomes researchquestions. This may include increasing knowledge aboutthe underlying philosophy and history of MMR, examplesof MMR research designs and principles, and the pros andcons of this approach above a purely qualitative or quanti-tative inquiry. To support this journey we provide a list ofrecommended texts which can form a starting point forthe curious researcher.Second, an open dialogue and collaboration betweenhealth outcome researchers with positivist or interpretivistleanings should be encouraged to prepare the ground forrobust MMR. In this context, it will be possible to designhealth outcomes research studies in which the whole isgreater than the sum of its parts and allow the researchcommunity to further the science through providinginnovative solutions to our research challenges.AbbreviationsFDA: Food and Drug Administration; ISOQOL: International Society for Qualityof Life Research; MMR: Mixed Methods Research; PRO: Patient-ReportedOutcomes; SIG: Special Interest GroupAcknowledgementsThis paper was reviewed and endorsed by the membership of theInternational Society for Quality of Life Research (ISOQOL) Mixed MethodsResearch Special Interest Group. As part of this review, insightful commentsfrom Benoit Arnould, Sophie Cleanthous, Chinmay Deshpande, and ClaudiaMarcela Vélez were implemented.This paper was reviewed and endorsed by the International Society forQuality of Life Research (ISOQOL) Board of Directors as an ISOQOLpublication and does not reflect an endorsement of the ISOQOLmembership.Regnault et al. Journal of Patient-Reported Outcomes  (2018) 2:19 Page 3 of 4Authors’ contributionsAR, TW and SB led the preparation of this manuscript on behalf of theInternational Society for Quality of Life Research (ISOQOL) Mixed MethodsResearch Special Interest Group (MMR SIG). AR and TW submitted theoriginal idea to the MMR SIG. AR, TW and SB drafted the manuscript andcoordinated the review by the MMR SIG membership and by the ISOQOLBoard of Directors. All reviewed and approved the final version.Ethics approval and consent to participateNACompeting interestsAR is an employee of Modus Outcomes. TW and SB declare that they haveno conflict of interest.Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.Author details1Modus Outcomes, 61, Cours de la Liberté, 69003 Lyon, France. 2RocheProducts Ltd., Welwyn Garden City, UK. 3The University of British Columbia,Vancouver, Canada.Received: 8 December 2017 Accepted: 3 April 2018ReferencesKey: *Recommended reading for researchers interested in MMR1. Denzin D. Moments, Mixed Methods, and Paradigm Dialogs. QualitativeInquiry. 2010;16(6):419–427.*2. Creswell, J. W., & Designing, P. C. V. L. (2007). Conducting mixed methodsresearch. Thousand Oaks, CA: Sage Publications *.3. Tashakkori A, Teddlie C. Handbook of mixed Methods in social behavioralresearch. Thousand Oakc,CA: Sage Publications; 2003.*.4. Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research.Thousand Oaks, CA: Sage Publications *.5. Wiering, B., de Boer, D., & Delnoij, D. (2017). Patient involvement in thedevelopment of patient-reported outcome measures: A scoping review.Health Expect, 20(1), 11–23.6. Special Section on Mixed Methods Research. (2012). Qual Life Research,21(3), 1573–2649.7. Roadmap to Patient-Focused Outcome Measurement in ClinicalTrials: US Food and Drug Administration; 2015. Available from:https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/ucm370177.htm.8. Tashakkori, A., & Editorial, C. J. W. (2007). The new era of mixed methods.Journal of Mixed Methods Research, 1, 1–6.9. Busija L, Nicholson G, Toombs M, Cinelli R, Easton C, Sanders K, et al.Developing Conceptual Model of the Role of Elders in the Wellbeing ofAustralian Indigenous Communities: Group Concept Mapping Study 23rdISOQOL Annual conference; Copenhagen, Denmark2016.10. Rosas, S. R., & Ridings, J. W. (2017). The use of concept mapping inmeasurement development and evaluation: Application and futuredirections. Eval Program Plann, 60, 265–276.11. Willgoss T. A Novel, Patient-Centered, Mixed Methods Approach to IdentifyingRelevant Concepts for Patient-Reported Outcome Measures and FacilitatingParticipant Engagement. 21st ISOQOL Annual Conference; Berlin,Germany2014.12. Barbic S. Development and testing of the Personal Recovery OutcomeMeasure (PROM) for people with mental illness: Application of Mixedmethods. 23rd ISOQOL Annual conference; Copenhagen, Denmark2016.13. Blum SI, Bushnell DM, McCarrier KP, Martin ML, Cano S, Liedgens H,et al. The Pain Assessment for Lower Back Symptoms (PAL-S) andImpacts (PAL-I): A Case Example of the Application of Mixed-Methods inPRO Instrument Development. 23rd ISOQOL Annual Conference;Copenhagen, Denmark2016.14. Cleanthous S. Why we should move Mixed Methods in PsychometricResearch from a three-step to a “two-step”. 23rd ISOQOL AnnualConference; Copenhagen, Denmark2016.15. Hudgens S. Application of mixed models for clinician reported outcomedevelopment 23rd ISOQOL annual conference; Copenhagen, Denmark2016.16. Fofana F, Bonnaud-Antignac A, Regnault A. A mixed method approach tohelp demonstrate saturation in qualitative research: applying Partial LeastSquare regression to qualitative data. 1st Mixed Methods InternationalResearch Association (MMIRA) Annual Conference; Boston, MA2014.17. Onwuegbuzie, A. J., Bustamante, R. M., & Nelson, J. A. (2010). Mixed researchas a tool for developing quantitative instruments. Journal of Mixed MethodsResearch., 4(1), 56–78 *.18. Gelhorn, H. L., Kulke, M. H., O'Dorisio, T., Yang, Q. M., Jackson, J., Jackson, S.,et al. (2016). Patient-reported symptom experiences in patients withcarcinoid syndrome after participation in a study of Telotristat Etiprate: Aqualitative interview approach. Clin Ther, 38(4), 759–768.19. Marrel A, Fofana F, Guillemin I. Increasing the interpretability of patient-reported outcomes questionnaire findings using a mixed methods design:An example in a rare cardiac clinical trial. 19th ISPOR Annual EuropeanCongress; Vienna, Autria2016.20. Patient-Centered Outcome Measures Initiatives in the Field of Rare Diseases.International Rare Diseases Research Consortium; 2016.21. Kirwan, J. R., de Wit, M., Frank, L., Haywood, K. L., Salek, S., Brace-McDonnell,S., et al. (2017). Emerging guidelines for patient engagement in research.Value Health, 20(3), 481–486.22. Maxwell, J., & Mittapalli, K. (2010). Realism as a stance for mixed methodsresearch. In A. Tashakkori & C. Teddlie (Eds.), SAGE handbook of mixedmethods in social & behavioral research (pp. 145–168). Thousand Oaks, CA:SAGE Publications Ltd.https://doi.org/10.4135/9781506335193.Regnault et al. Journal of Patient-Reported Outcomes  (2018) 2:19 Page 4 of 4


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items