UBC Faculty Research and Publications

Trial-based clinical and economic analyses: the unhelpful quest for conformity Whitehurst, David G; Bryan, Stirling Dec 5, 2013

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-13063_2013_Article_1859.pdf [ 178.43kB ]
JSON: 52383-1.0223366.json
JSON-LD: 52383-1.0223366-ld.json
RDF/XML (Pretty): 52383-1.0223366-rdf.xml
RDF/JSON: 52383-1.0223366-rdf.json
Turtle: 52383-1.0223366-turtle.txt
N-Triples: 52383-1.0223366-rdf-ntriples.txt
Original Record: 52383-1.0223366-source.json
Full Text

Full Text

COMMENTARY Open AccessTrial-based clinical and economic analyses: theunhelpful quest for conformityDavid GT Whitehurst1,2* and Stirling Bryan2,3AbstractWhere there is conformity across the findings, interpretation and implications of ‘clinical’ and ‘economic’ research,there is limited cause for concern. However, there is often unease when apparent contradictory conclusions aredrawn from the same study. Given the ever increasing role for economic evaluation in healthcare decision making,this commentary challenges the necessity of compatibility between clinical and economic evaluation.Keywords: Randomized controlled trial, Economic evaluation, Outcomes measurement, Cost-utility analysis,Scientific paradigms, Estimation, InferenceBackgroundWith many countries requiring evidence of cost-effec-tiveness prior to passing judgment on the value of amedical intervention, it is inevitable that clinical andeconomic evidence will be considered concurrently bypolicy makers [1,2]. Such an approach enables decisionmakers to reflect on scientific and social value judg-ments (addressing both equity and efficiency) when allo-cating scarce resources [3]. In circumstances wherethere is conformity across the findings and implicationsof clinical and economic research, few concerns areraised. However, there is often unease when apparentcontradictory conclusions are drawn within a study, ei-ther because the clinical research shows benefit but theeconomics indicates that a therapeutically beneficialintervention is not cost-effective [4,5], or the clinical re-search shows little benefit but the economics reveals evi-dence of cost-effectiveness for one of the interventionsbeing evaluated [6]. For example, a recent study reportedthat a lifestyle program for adults at risk for type 2 dia-betes and/or cardiovascular disease was not more effec-tive in reducing these risks when compared to generalhealth brochures, although the intervention had a highprobability of being cost-effective [7].The former situation is a reflection of the unavoidableneed for healthcare decision-making within the con-straints of a finite budget; sometimes, health benefits at-tributable to an intervention may not be sufficient towarrant associated increases in expenditure. Unease inthe latter situation relates, primarily, to the differentmethods of analysis adopted in clinical and economicevaluation. Differing opinions regarding the role of eco-nomic evaluation alongside randomized controlled trials(RCTs) with indeterminate clinical findings can result indifficulties/delays in publishing economic evaluation re-sults, irrespective of whether concerns are expressedwithin the study team or during the peer-review pro-cess. In this commentary, we challenge the necessityof compatibility between clinical and economic trial-based research.Clinical and economic evaluation: distinctscientific paradigmsThomas Kuhn, the American physicist and philosopher,wrote extensively on the history of science and, in par-ticular, on the notion of paradigms in science. Describinga paradigm as ‘the entire constellation of beliefs, values,techniques, and so on shared by the members of a givencommunity’ [8], Kuhn’s definition provides an ideal back-drop to revisit current, widely-practiced clinical and eco-nomic evaluative frameworks, that is, pragmatic RCTs andtrial-based cost-effectiveness analyses.There are stark differences between clinical and eco-nomic evaluation with respect to the purpose of research* Correspondence: david_whitehurst@sfu.ca1Faculty of Health Sciences, Blusson Hall 10504, Simon Fraser University,8888 University Drive, Burnaby, British Columbia V5A 1S6, Canada2Centre for Clinical Epidemiology & Evaluation, Vancouver Coastal HealthResearch Institute, 7th Floor 828 West 10th Avenue, Vancouver, BritishColumbia V5Z 1M9, CanadaFull list of author information is available at the end of the articleTRIALS© 2013 Whitehurst and Bryan; licensee BioMed Central Ltd. This is an open access article distributed under the terms of theCreative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use,distribution, and reproduction in any medium, provided the original work is properly cited.Whitehurst and Bryan Trials 2013, 14:421http://www.trialsjournal.com/content/14/1/421(‘belief ’) and the adopted analytic methodologies (‘tech-nique’). Differences in belief, in this context, exist in therespective objectives of clinical intervention studies andeconomic evaluation. The pragmatic RCT is conductedto quantify the magnitude of some factor(s) of interest,providing an estimate of effectiveness for an interventioncompared to an appropriate control group. The purposeis to determine the ability of an intervention to improvepatient-level outcome(s) (or, in the case of noninferioritytrials, to test whether an intervention is not unaccept-ably worse than a current procedure or treatment) andthe contribution of the research lies in the increased evi-dence base for clinicians and researchers. Alternatively,the purpose of cost-effectiveness analysis is to providedecision-makers with evidence on the value of an in-tervention, reflecting efficiency and equity [3]. Thisevidence is then considered against competing claimsfor healthcare resources across a multitude of medicalconditions.Despite fundamental differences between clinical andeconomic evaluation, the distinction is somewhat blurredwith respect to analytic methodologies, despite the avail-ability of research methods and reporting guidelines fortrial-based economic analyses [9]. As an illustration, manyreaders will have read or heard (or said) comments suchas, ‘economic analysis is only important if there is evidenceof clinical benefits.’ In our opinion, this is wholly inap-propriate as a starting point for consideration of cost-effectiveness. Demonstrable clinical benefit is not aprerequisite for economic evaluation; large clinical bene-fits may be too expensive, small clinical benefits may becost-effective and, dare we say, health decrements may beassociated with sufficient cost savings to warrant an inter-vention becoming part of routine care. We use the words'large' and 'small', as opposed to 'significant' and 'non-sig-nificant', to emphasize the difference in analytic focus be-tween conventional economic evaluation (estimation) andclinical evaluation (inference), which has been discussedelsewhere [9-12]. We return to the questionable useof inferential statistics for economic evaluation in thefollowing section.Given the different research objectives of RCTs andeconomic evaluations, it is not surprising that differencesexist with regard to the measurement of patient-levelbenefit (‘value’). Clinical relevance is the key determinantwhen selecting the primary outcome in a clinical trial [13];the chosen outcome measure is required to measure pa-tients’ response to treatment, often focusing on a condi-tion-specific measure of response that is meaningful tothe clinical community, with the results being interpret-able at the individual patient level. However, the relevanceof the chosen clinical outcome measure is likely neg-ligible when exploring the ‘economic’ consequences ofa particular intervention. For example, assessment ofcost-effectiveness within a cost-utility framework bringstogether two components: costs (defined by the pers-pective of the analysis) and quality-adjusted life years(QALYs). The standard approach for estimating QALYsprovides for the incorporation of societal health statevalues through the use of preference-based health-relatedquality of life (HRQoL) measures [1,2].The results of an economic evaluation only havemeaning in a comparative sense. Although clinical effect-iveness can be explored within the boundaries of a singlestudy, value judgments regarding efficiency and equityrequire consideration of other demands on healthcareresources. The pursuit of this broader, societal objec-tive - albeit with patient-level data - is a reflectionthat society comprises individuals, and all individualsare potential patients.Confusing analytic conciliationThe health economics research community has providedits own contribution to the evaluative confusion. In par-ticular, with respect to the measurement of health bene-fit, studies published in recent years have subjectedpreference-based HRQoL measures to concepts that arefundamentally inferential in nature; namely, the minim-ally important difference (MID) and the noninferioritymargin [14-16].The MID is ‘the smallest difference in score in the do-main of interest which patients perceive as beneficialand which would mandate, in the absence of trouble-some side effects and excessive cost, a change in the pa-tient’s management’ [17]. Although the concept of theMID is well accepted in clinical research, we believethere is some concern regarding its application for cost-effectiveness research (despite a number of studies havingestimated MIDs for preference-based HRQoL instruments[15,16]). MIDs, however defined, are specific to a singleoutcome measure and are interpreted as being applicableat a patient-level, meaning that the MID concept is toonarrow to be meaningful in economic evaluation. Cost-effectiveness estimation requires the simultaneous con-sideration of costs and effects. The magnitude of a cost(effect) difference, viewed in isolation, holds limited valueuntil combined with the respective difference in effects(costs). Just because it is possible to construct a minimallyimportant difference for preference-based measures doesnot mean that it is useful.Similar issues arise in relation to noninferiority trials[14]. Guidelines for the conduct of economic evaluationalongside such trials state that acceptable differences incosts and effects must be defined a priori in order toexplore non-inferior or equivalent cost-effectiveness ofstudy treatments. However, these requirements take noaccount of the key analytic focus of economic evalu-ation: estimation of the joint density of cost and effectWhitehurst and Bryan Trials 2013, 14:421 Page 2 of 3http://www.trialsjournal.com/content/14/1/421differences [11]. Determining whether a cost differenceor QALY difference is ‘acceptable’ is not the role of theanalyst. In a trial-based economic evaluation, irrespectiveof the clinical findings or RCT design (superiority, nonin-feriority, or equivalence), an analyst should focus on esti-mating cost and effect differences and quantifying thelikelihood that an intervention is cost-effective [9,12].ConclusionsThere is a need to understand better the fundamentaldifferences in the questions being addressed by clinicaland economic evaluation performed alongside RCTs,and for all researchers to be comfortable with apparentlydiscordant findings. It is our hope that this commentarywill provide clarity, or spark further debate, and afford areference point for future discussions about ‘incompa-tible’ RCT findings.AbbreviationsHRQoL: Health-related quality of life; MID: Minimally important difference;QALY: Quality-adjusted life year; RCT: Randomized controlled trial.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsBoth authors were involved in the conception and framing of theCommentary, and have read and approved the final manuscript. DGTWwrote the first draft and subsequent revisions. SB was involved in review andcritical revision of the content prior to submission.Authors’ informationThis paper draws on the experiences of both authors in the field ofeconomic evaluation, covering methodological and applied research projectsacross a range of clinical specialties. More specifically, the conception of thisCommentary relates to numerous conversations with colleagues about thedistinctions between clinical and economic evaluation, and the frequentneed to appease peer reviewers’ concerns about apparent mixed messageswithin a single study.AcknowledgementsThis research received no specific grant from any funding agency in thepublic, commercial, or not-for-profit sectors. We would like to thank the re-viewers for their constructive comments.Author details1Faculty of Health Sciences, Blusson Hall 10504, Simon Fraser University,8888 University Drive, Burnaby, British Columbia V5A 1S6, Canada. 2Centre forClinical Epidemiology & Evaluation, Vancouver Coastal Health ResearchInstitute, 7th Floor 828 West 10th Avenue, Vancouver, British Columbia V5Z1M9, Canada. 3School of Population & Public Health, University of BritishColumbia, 2206 East Mall, Vancouver, British Columbia V6T 1Z3, Canada.Received: 16 August 2013 Accepted: 21 November 2013Published: 5 December 2013References1. Pharmaceutical Benefits Scheme (PBS): Guidelines for Preparing Submissionsto the Pharmaceutical Benefits Advisory Committee (PBAC) (Version 4.3).Canberra, Australia: Pharmaceutical Benefits Scheme; 2008.2. National Institute for Health and Care Excellence: Guide to the Methods ofTechnology Appraisal 2013. London, UK: National Institute for Health andCare Excellence; 2013.3. Rawlins MD, Culyer AJ: National institute for clinical excellence and itsvalue judgments. BMJ 2004, 329:224–227.4. MacPherson H, Tilbrook H, Bland JM, Bloor K, Brabyn S, Cox H, Kang’ombeAR, Man MS, Stuardi T, Torgerson D, Watt I, Whorwell P: Acupuncture forirritable bowel syndrome: primary care based pragmatic randomisedcontrolled trial. BMC Gastroenterol 2012, 12:150.5. Stamuli E, Bloor K, MacPherson H, Tilbrook H, Stuardi T, Brabyn S, Torgerson D:Cost-effectiveness of acupuncture for irritable bowel syndrome: findingsfrom an economic evaluation conducted alongside a pragmaticrandomised controlled trial in primary care. BMC Gastroenterol 2012, 12:149.6. Whitehurst DG, Lewis M, Yao GL, Bryan S, Raftery JP, Mullis R, Hay EM: Abrief pain management program compared with physical therapy forlow back pain: results from an economic analysis alongside arandomized clinical trial. Arthritis Rheum 2007, 57:466–473.7. van Wier MF, Lakerveld J, Bot SD, Chinapaw MJ, Nijpels G, van Tulder MW:Economic evaluation of a lifestyle intervention in primary care toprevent type 2 diabetes mellitus and cardiovascular diseases: arandomized controlled trial. BMC Fam Pract 2013, 14:45.8. Kuhn TS: The Structure of Scientific Revolutions. 2nd edition. Chicago:University of Chicago Press; 1970.9. Petrou S, Gray A: Economic evaluation alongside randomised controlledtrials: design, conduct, analysis, and reporting. BMJ 2011, 342:d1548.10. Claxton K: The irrelevance of inference: a decision-making approach tothe stochastic evaluation of health care technologies. J Health Econ 1999,18:341–364.11. Briggs AH, O’Brien BJ: The death of cost-minimization analysis? HealthEcon 2001, 10:179–184.12. Briggs A: Economic evaluation and clinical trials: size matters. BMJ 2000,321:1362.13. Gebski V, Marschner I, Keech AC: Specifying objectives and outcomes forclinical trials. Med J Aust 2002, 176:491–492.14. Bosmans JE, de Bruijne MC, van Hout HP, Hermens ML, Adèr HJ,van Tulder MW: Practical guidelines for economic evaluations alongsideequivalence trials. Value Health 2008, 11:251–258.15. Lee BB, King MT, Simpson JM, Haran MJ, Stockler MR, Marial O, Salkeld G:Validity, responsiveness, and minimal important difference for the SF-6Dhealth utility scale in a spinal cord injured population. Value Health 2008,11:680–688.16. Luo N, Johnson J, Coons SJ: Using instrument-defined health statetransitions to estimate minimally important differences for fourpreference-based health-related quality of life instruments. Med Care2010, 48:365–371.17. Jaeschke R, Singer J, Guyatt GH: Measurement of health status.Ascertaining the minimal clinically important difference. Control Clin Trials1989, 10:407–415.doi:10.1186/1745-6215-14-421Cite this article as: Whitehurst and Bryan: Trial-based clinical andeconomic analyses: the unhelpful quest for conformity. Trials2013 14:421.Submit your next manuscript to BioMed Centraland take full advantage of: • Convenient online submission• Thorough peer review• No space constraints or color figure charges• Immediate publication on acceptance• Inclusion in PubMed, CAS, Scopus and Google Scholar• Research which is freely available for redistributionSubmit your manuscript at www.biomedcentral.com/submitWhitehurst and Bryan Trials 2013, 14:421 Page 3 of 3http://www.trialsjournal.com/content/14/1/421


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items