Open Collections

UBC Faculty Research and Publications

Rules of scientific inquiry and clinical trials : improvement is needed Kazanjian, Arminée, 1947-; Hadorn, David C., 1952-; Green, C. J. (Carolyn Joanne), 1956- Feb 28, 1997

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Kazanjian_A_et_al_Scientific_enquiry.pdf [ 7.35MB ]
JSON: 52383-1.0045269.json
JSON-LD: 52383-1.0045269-ld.json
RDF/XML (Pretty): 52383-1.0045269-rdf.xml
RDF/JSON: 52383-1.0045269-rdf.json
Turtle: 52383-1.0045269-turtle.txt
N-Triples: 52383-1.0045269-rdf-ntriples.txt
Original Record: 52383-1.0045269-source.json
Full Text

Full Text

Submitted for Publication - Do not quote, reproduce or circulateCentre for Health Servicesand Policy ResearchRULES OF SCIENTIFIC INQUIRY ANDRANDOMIZED CONTROLLED TRIALSArmlnee KazanjianDavid HadornCarolyn J. GreenBCOHTA 97:1D FEBRUARY 1997B.C. Office of Health Technology AssessmentDiscussion Paper SeriesTHE UNIVERSITY OF BRITISH COLUMBIARules of scientific inquiry and clinical trials:Improvement is neededArminee Kazanjian, Dr. Soc.David C. Hadorn, M.D., M.A.Carolyn J. Green, B.H.Sc.(p.T.), l\~.Sc.British Columbia Office of Healtb Tecbnology AssessmentTbe University of Britisb ColumbiaSources of Support: British Columbia Ministry of Health grant through the Office of theCoordinator of Health Sciences, The University of British ColumbiaAutbor Responsible For Correspondence:Dr. Arminee Kazanjian, Associate DirectorCentre for Health Services and Policy ResearchThe University of British Columbia429-2194 Health Sciences MallVancouver, British Columbia V6T IZ3 CanadaTel: (604) 822-4618 Fax: (604) 822-5690AbstractValid scientific research on the effectiveness of interventions provides the foundation forimproving the quality of health care. The social nature of this scientific activity requires acommitment on the part of scientists to follow established rules of inquiry in order to fosterobjectivity in observation and conclusion. The existence and general acceptance of these rules isin large part responsible for the privileged position awarded to science by contemporary Westernsocieties,When scientists do not follow prescribed rules, the validity of their observations, studies,and conclusions is compromised. Such rule infractions are not always acknowledged by therelevant scientific community, in which case society at large remains unaware of the problem.We illustrate this problem by reference to the use (or non-use) of established rulesdesigned to minimize observer bias in clinical trials and ensure unbiased reporting-areas whichrequire subjective judgment and which are therefore modifiable. To the extent that investigatorshave strayed from these rules, the affected trials should be considered less than fully scientific andsociety should hesitate to accept their results. The identification of breaches of scientific rules ofinquiry through rigorous critical appraisal protects the public from recommendations emanatingfrom flawed science.KeywordsResearch Standards; Experimental Design; Observer BiasBe Office ofHealth Technology AssessmentRules ofscientific inquiry2Social context of scientific knowledgeIt is well-accepted by philosophers and historians of science that science is a "socialenterprise."(1) Although many concepts are intended by this characterization, perhaps mostfundamental is the tenet that observations, studies, and experiments gain scientific status by virtueof following rules ofinquirysubscribed to by the relevant community of scientists.As noted by Helen Longino in her book Scienceas Social Knowledge:One does not simply declare oneself a biologist but learns the traditions, questions,mathematical and observational techniques, 'the sense of what to do next,' from someonewho has herself or himself been through a comparable initiation and then practiced. One'enters a world' and learns how to live there. (2 p.67)Such initiations also occur among physicians, as noted by Robert Merton:The profession ofmedicine ... has its own normative subculture, a body of shared andtransmitted ideas, values and standards toward which members of the profession areexpected to orient their behaviour. (3 p.71)Indeed, following accepted roles and techniques is one of the hallmarks of science.To a substantial extent the rules and standards of scientific practice act to ensureobjectivity in results emanating from the various forms of scientific inquiry. (4) Unlike theproducts of artistic endeavors (e.g., paintings, books, poetry), the value and validity of scientificactivities is not (or ought not be) "in the eye of the beholder." Only to the extent that the tools andBe Office ofHealth Technology AssessmentRules ofscien/iRc inquiry3procedures of inquiry can, in principle, be wielded by any suitably skilled and equipped individual,and only to the extent those tools and procedures produce the same (or similar) results whenwielded in the same (or similar) settings, can the products of inquiry be considered objective--andthus scientific.Similarly, criticisms of seemingly scientific activities must be based on accepted standardsof inquiry:In order for criticism to be relevant to a position it must appeal to something accepted bythose who hold the position criticized.... This cannot occur at the whim of individualsbut must be a function of public standards or criteria to which members of the scientificcommunity are or feel themselves bound. These standards can include both substantiveprinciples and epistemic, as well as social, values .... [I]t is the existence of standards thatmakes the individual members of a scientific community responsible to something besidesthemselves. (2 p.77)Observation as scientific activityThe most fundamental of all scientific processes is observation. As with all scientificactivities, to count as scientific, observations must be made in accordance with accepted rules andstandards. As noted by Peter Kosso in his book, Reading the Book ofNature:Science cannot accept just any old report of the senses. As a rule, observations must becarefully done and repeatable. They must be carried out in a controlled way and under theBe Office ofHealth Technology AssessmentRules ofscienUtic inquiry4proper conditions.. . . Responsible observing is observing we are able to justify. Thisrequires an understanding of the relevant conditions under which an observation is done... . [T]he scientific community must understand what it takes to do the observationproperly if the observation is to be accepted as scientific evidence. Science, after all, is apublic enterprise, and the matters ofjustification are matters of community. (5 p.112)As always, the rules of observation strive to ensure the objectiveness of observations(technically, of observation reports), in the sense described above. The objectiveness ofobservation reports consists of two dimensions: (1) protection against bias (such as throughensuring reproducibility of observations by multiple independent observers) and (2) independenceof the theoretical basis of observation from the theory tested by observation.Regarding protec~on against bias, scientists, like everyone, tend to observe what theyhope or expect to observe. So much is simple human nature, but the complications wrought bysuch biased observation (and observation reports) can be harmful. Bias need not be conscious(although this occurs often enough); indeed, the more pervasive problem is due to unconsciousbias.The existence of unconscious bias in scientific.observation was perhaps first documentedrigorously by Robert Rosenthal and co-workers in the late 1950's and early 1960's. (6 7 8) Theseinvestigators used a series of paradigms in which experimenter-subjects were led to believe, priorto testing the performance of experimental subjects on various cognitive tasks, that their subjectswould perform either better or worse than average. To a substantial and disturbing effect, theseexpectations were found to be fulfilled, despite the absence of any actual differences in subjects'abilities.Be Office ofHealth Technology AssessmentRules ofscienli5c inquiry5Subsequent to these pioneering experiments, an entire literature has accumulateddocumenting the biasing effect of self-fulfilling prophecy, (9 10 11) including further researchinto the reasons students perform to teachers' expectations, (12 13) evaluation of workplaceproductivity, (14) and the effects of clinicians' initial expectancies on diagnosis and treatment.(15)In the more exact domains of science, such as chemistry, protection against bias isachieved by means of measurement instruments (e.g., thermometers, spectroscopes) whosereadings are readily reproducible and accessible to anyone. In less exact sciences, such as clinicalresearch, protection against bias is achieved by the use of additional measurement techniques suchas blinding individuals charged with making critical observations (e.g., ensuring observers ofpatient outcome in clinical trials are unaware of which treatment was applied).With respect to the second tenet of objectiveness in scientific observation, it is essentialthat the theory upon which the credibility of a set of observations is based is independent of thetheory whose truth (or empirical adequacy) is being tested by those observations.For example, expansion of a mercury column inside a thermometer provides independentevidence in support of the theory that mixing an acid and a base produces heat because our theorylinking heat to mercury expansion is distinct from our theory about acids and bases. By contrast,following Kosso, evidence obtained from a "caloric flow meter" (a machine designed to detect theflow of"caloric fluid"-formerly hypothesized to explain the phenomenon of heat) would fail thetest of independence if used to prove the existence of caloric fluid:[T]he problem is not that the observation is very indirect. Lots of good observationalevidence for lots of good theories is very indirect. Nor is the problem that the observationBe Office ofHealth Technology AssessmentRules 01sdenti5c inquiry6is theory-laden. All observation is theory-laden. It's not even that it is laden with a weak,unproven theory.... The problem is that the observation is laden with caloric theory. Asevidence for caloric theory, then, it fails on evaluation of independence. It is not objectiveevidence... .Independent evidence is objective evidence, and the requirement ofindependence is a key ingredient of the scientific process that prevents problematiccircularity in the justification of theories. (5 p.l72-173; 158)To summarize, scientific disciplines develop relevant rules of observation according to thespecific goals of those disciplines and specific threats to objectiveness. In experimental inquiry,researchers seek to enhance validity by minimizing bias and striving to separate the theoreticalfoundation of the observation from the theory under study.Bias in clinical trialsIn the realm of clinical research, the problem of observer bias is most severe when, as is often thecase, observer-investigators have clear prior opinions about the effects of treatments under study.If observers already suspect that Treatment A is better than Treatment B, they are likely to observemore improvement in patients receiving Treatment A than in those receiving Treatment B. Priorbeliefs about the relative effectiveness of treatments under study in a clinical trial are particularlyproblematic when, as is often the case, trials are used to "confirm" the benefits of treatmentsalready in widespread use. Even for new interventions, clinical studies are usually undertaken toestablish efficacy, not inefficacy. Rules of inquiry protect a trial from the introduction of bias.Be Office ofHealth Technology AssessmentRules ofscienli5c inquiry7Bias may be variously defined as a systematically wrong estimate of a parameter ofinterest, the amount by which an estimate differs from the true value, or any process tending tolead to results differing systematically from the truth.(16 p. 176)In order to provide assurance to those outside the clinical research community -practitioners, payers policy-makers - that rules of scientific inquiry have been adhered to, anumber of features of clinical trials are routinely assessed through systematic review and criticalappraisal.. Some of these rules are often enforced by research funding agencies and publication~.For example, random allocation, a technique which ensures that in the long run treatment andcontrol subjects are comparable in relevant ways, is considered so important for controllingselection bias that non-randomized trial designs are considered fatally flawed by many observers.Although important, we do not address the issue of allocation bias in this paper. Rather,we focus on the problem of observerbias with respect to the observations (and observationreports) upon which inferences are drawn concerning treatment effectiveness (or lack thereot).As it turns out, rules designed to ensure unbiased complete reporting of outcomes are not asstringently enforced as those pertaining to randomized allocation. The neglect of these features isunfortunate, because both sets of rules are equally likely to render trial results invalid.Furthermore, adherence to rules of observation does not represent an undue burden on trialists,especially compared to the expense and logistical difficulties involved in conducting a randomizedtrial. Insisting that randomized clinical trials adhere to these rules is both appropriate andnecessary if the results of the trial are to count as scientific.BeOffice ofHealth Technology AssessmentRules ofscientiRc inquiry8Blinding of outcome assessmentTo guard against observation bias, rules of inquiry for clinical research have beendeveloped and promulgated in textbooks of epidemiology and clinical research.Specifically, it is widely held by scientists in these disciplines that individuals makingobservations of relevant treatment outcomes should be blinded or masked with respect tothe treatment received. (17 18 19 20 21) That is, they must not know if the patient has received,for example, Treatment A or Treatment B. Schulz et al(22) found that unblinded studies reportedsignificantly higher estimates of treatment effectiveness than blinded trials on the same topic.Blinding helps to achieve objectivity both by reducing bias and by ensuring independenceof the theories underlying observations from the theory being tested. With respect to the latteraim, as noted by Kosso, "an understanding of the causal interaction between the object and theviewer" (5 p.l64) is needed to assess independence of evidence. In the case of unblindedobservers, this causal interaction includes elements of the theory under test. The theory is "in" theobserver, so to speak, and thus in the observation. Such "contaminated" observations do not,therefore, constitute independent evidence of the theory under test. As such, they do not meet thetest of objectivity required of scientific evidence.While this conclusion may seem overly harsh to many investigators, who will protest thatmany kinds of clinical trials simply "cannot" be performed blind, it is always possible to blind theassessors of key clinical outcomes. This point was made by Thomas Chalmers:Many people do not realize that in situations where the physician caring for the patientcannot be blinded, it is still possible to blind some observers who are gathering endpointBe OfIice ofHealth Technology AssessmentRules ofscientific inquity9data. In other words, the physicians who are making the critical decisions on whether onetherapy is better or worse than another can be blinded, even though those taking care ofthe patient may not be. (23 p.139)Thus, it is insufficient to claim simply that blinded assessment is "too hard" or "notpossible" in some clinical trials. In this same paper Chalmers noted that only 64% of a series of300 randomized trials reviewed by him and co-workers had been conducted using properly blindedassessment of outcomes.Blinding of data analysis and reportingMore rigorous scientific standards may also be applied to data analysis and reporting, bothprocesses with a subjective component and therefore prone to investigator bias. Although rare,Gotzsche found grounds for blind analysis and reporting while conducting a meta-analysis ofnonsteroidal, antiinflammatory drugs (NSAIDS). Recalculation of p-values in instances wherethis was possible revealed 12 double-blind trials with miscalculations-all errors favored the newdrug over the 01d.24 Gotzsche subsequently concluded:Blinding during data analysis and writing of manuscripts may be important and shouldtherefore be assured by, for example, letting an independent office or agency perform therandomization and hold the randomization codes .... [S]ham codes would allow thestatistical analysis to be made and two manuscripts to be written, approved by allcoauthors and filed with the office, before the code is broken.{2S p.289)Be Office ofHealth Technology AssessmentRules ofsdenbnc inquily10Blinding this final step, in the clinical trial process, fulfills conditions of objectivity at themost stringent level and ensures independence of the evidence.Inter-observer reliabilityAnother major component of the requisite objectivity of scientific observation is thereliability of observation-that is, the extent to which different observers arrive at the same (orsimilar) judgments concerning the observation of interest. Reliability of measurement is aprerequisite for validity. Indeed, inter-observer reliability is a prerequisite for measurement. Onlywhen independent observers are able to arrive at the same (or similar) determinations of thequantity involved (e.g., temperature, briskness of deep tendon reflexes) can an attribute beconsidered measurable. For example, the measurement of temperature in a chemistry experimentis generally considered highly reliable (and therefore valid) because multiple observers are able totake independent readings from and of several standard thermometers which are in closeagreement with each other.The observations used in clinical trials are seldom amenable to such high degrees of inter-observer agreement, of course. With the notable exception ofdeath, in which case inter-observervariation is not much of an issue (unless the cause of death must be specified), almost all clinicaloutcomes are susceptible to inter-observer variation. Even "precise" tests such as X-rays andblood pressure readings are subject to substantial inter-observer variation. A large literature hasaccumulated on this topic. (26 27 28 29)The lack of precision in measurement implied by large inter-observer variation has theeffect of increasing random (and standard) error and producing less precise effect estimates. InBe Office ofHealth Technology AssessmentRules ofscien/i6c inquiry11other circumstances, large inter-observer variability may compound bias introduced by unblindedobservers. In trials dependent on "soft" endpoints known to have substantial inter-observervariability, such as "quality of life" or the "presence or absence ofasymmetries in muscle strengthor in deep tendon reflexes", inter-observer reliability testing is especially critical.<30-32) In thesecircumstances, when protection against bias, as afforded by blinding, is not in place; the low inter-observer variability compounds the biases inherent in pre-conceived notions of effectiveness byreducing the validity of the measurements in terms of which outcomes are assessed and reported.Agreement among independent observers ensures that the measures are replicable and enhancesthe validity of the clinical observation.Because of this generic problem, clinical investigators should provide evidenceconcerning the inter-observer reproducibility of the outcome assessments used in RCTs,particularly in cases of unblinded trials. This is not to say that everypatient must be assessed bytwo or more independent observers, of course, but rather that inter-observer reliability should betested in an appropriate subset of study patients.Reporting of resultsThe final rule or methodological standard to be considered here concerns the reporting ofobservations. Specifically, observations should be reported in a manner that provides the mostaccurate indication of the significance ofthe results. In the hard sciences this tenet is rarelyproblematic, but in the softer sciences, including medicine, there is often great discretion availablein how observations are reported.With respect to reporting the results of clinical trials, for example, the most commonlyreported parameter by far is relative risk reduction, in which a treatment might be said, forBe Office ofHealth Technology AssessmentRules ofscienti5c inquiry12example, to reduce mortality by 50 percent when applied to some patient population. Proponentsof relative risk note that it is useful in applying the results of a trial to populations with differingbaseline risk.As is well known, (333435) however, relative risk reduction can be grossly misleading,as this statistic does not distinguish between, say, a reduction in mortality rates from 100 percentto 50 percent (a substantial effect by anyone's lights) versus a reduction from 4 percent to 2percent-s-an effect of dubious significance. For this reason, it is widely accepted that effectsobserved in clinical trials should be portrayed primarily in terms of absolute effect size especiallywhen relative effect size is reported in isolation from information about absolute effect size.A third measure of effect size has been identified, known as the Number Needed to Treat,or NNT, (= 1 / absolute effect size). (29) (For example, a treatment reducing mortality by 2percent would need to be-administered to 50 patients in order to extend one life.) Intuitively, itseems that this statistic provides an important, even critical insight into the practical significanceof the observed effect size. For this reason, Laupacis et al. recommended that "the relative riskreduction should not be cited without simultaneously indicating the absolute risk reduction or theNNT." (36 p.A14)Several studies have demonstrated that the degree of enthusiasm evinced by clinicians forthe results of studies depends substantially upon the mode in which those results are presented,with NNT figures resulting in the least enthusiasm. (28 29 30 37) As noted by the investigators inone such study:...we believe the most plausible explanation for the present results is that ~T's andsimilar measures dampen enthusiasm for drug therapy by offering a more clinicallyBe Office ofHealth Technology AssessmentRules ofscientilic inquiry13meaningful view of treatment effects ....We predict that summary measures other thanpercentage reductions in relative or absolute risk will emerge as a more relevant andethically grounded way to summarize and compare treatment effects for physicians,patients, and policy makers alike. (28 pp.919-920)A good example of how NNT can provide a less optimistic (or less dramatic) portrayal ofresearch results can be found in a recent meta-analysis of six RCTs on the effect ofwarfarin inpreventing stroke in patients with chronic non-valvular atrial fibrillation. (38) Collectively, theobservations from these trials implied a 3.1 percent absolute reduction in risk of stroke in treatedpatients (from 4.5 percent untreated to 1.4 percent treated).The investigators concluded that their analysis "has confirmed that warfarin dramaticallydecreases the risk of stroke by a relative risk reduction of 68 percent." This depiction of effectsize seems quite inflated when considered from the standpoint ofNNT:... the 3.1 percent absolute risk reduction in all strokes claimed in the pooled analysiscorresponds to an NNT of33 (1/3.1). That is, 33 patients would need to be treated withwarfarin in order to prevent one stroke. Of the remaining 32 patients, 31 would not have astroke even if left untreated (and thus do not need warfarin) and one patient would have astroke even if treated with warfarin (and thus would not benefit from warfarin). Thisperspective on the overall clinical yield of treatment is far more realistic and meaningfulthan the relative risk reduction figure 68 percent implies. (39 p.27)Be Office ofHealth Technology AssessmentRules ofscientific inquiry14Review of recent ReTsAlthough rules of scientific inquiry apply to many different experimental andobservational research designs, the randomized controlled trial is regarded as the gold standardand therefore is required to meet more stringent criteria for rigour. In order to determineprevailing practices with respect to the standards of observer blinding, inter-observer reliabilitytesting, and appropriate reporting of results, we reviewed all RCTs reported in the the Journal ofthe American Medical Association, the Lancet, and the British Medical Journal in 1996. Onlystudies reporting randomized allocation of treatment were included. We noted whether or notpatient assessment was conducted by observers blinded to treatment assignment and whether anyattempt was made to assess inter-observer reliability of the assessments. We also determinedwhether effect sizes were conveyed predominantly in terms ofNNT, absolute effect sizes orrelative effect sizes.Altogether, 118 RCTs were published by the threejoumals during 1996 (JAMA =30; BMf= 30; Lancet = 58). Of these, 6 I (51.7 percent) were conducted using blinded patient assessment.After eliminating the 11 studies that assessed outcomes only in terms of patient-completedquestionnaires, blood chemistries, or 'hard' endpoints (e.g., conception and live birth), 61 out of107 RCTs (57.0 percent) were conducted using blinded assessment. (Note that separate testing ofquestionnaire reliability and chemistry calibration was generally reported.)Eleven studies (9.3 percent) observed no effect with treatment. Of the remaining 107studies, 83 (78 percent) reported results in terms of absolute effect size. No study reported effectsizes in terms ofNNT as a primary effect measure.Only one study reported inter-observer reliability of patient assessment. An additionalthree studies conducted test-retest reliability of the patient questionnaires used to assess outcome.Be Office ofHealth Technology AssessmentRules ofscientific inquiry15DiscussionThe proportion ofRCTs using blinded assessment in our sample (57 percent) is less thanthe proportion reported by Chalmers et al. in 1983 (64 percent). Although we are unable to saywhether this represents a true decline over the intervening 13 years, certainly the situation does notappear to have improved since then.As discussed above, unblinded assessments will frequently be seriouslybiased-especially, as is often the case, when clinicians have an opinion about which treatment ismore effective. Under such circumstances, observations cannot be considered objective and thestudy cannot, therefore, be considered grounded in science.We were disappointed to find that only one trial conducted (or at any rate reported) theresults of inter-observer reliability testing with respect to outcome assessment. The combinationof unblinded observation and absence of reliability testing is worrisome, especially in the contextof 'soft' endpoints, such as neurological or behavioral deficits.Regarding the method of reporting study findings, we were again disappointed to find thatnone of the RCTs reported effect sizes in terms ofNumber Needed to Treat. Nine years after theadvantages of this statistic were first described, (29) clinical investigators have still not receivedthe message. On the other hand, about 80 percent of the studies described effects sizes primarilyin terms of absolute risk reduction, thus meeting the prevailing standard in this respect. Some ofthe remaining studies used statistical methods which virtually require reporting in terms of relativerisk reduction, including Cox proportional hazards and Kaplan-Meier survival curves. In view ofthe discussion above, we must question the appropriateness of these methods in the context ofsmall effect sizes where the lack of clinical significance is obscured.Be Office ofHealth Technology AssessmentRules ofscientific inquiry16If scientific inquiry is a social enterprise, we can only conclude that too many clinicalscientists are behaving in ways that do not benefit society. In particular, the apparent disregard forthe importance of blinding, inter-observer reliability testing, and appropriate reporting standardsmanifested by a substantial proportion of contemporary investigators is incommensurate withgood scientific practice and contrary to the public interest.We call on clinical investigators, funding agencies, peer-reviewed journals, and theacademic community at large to ensure adherence to appropriate t ules of inquiry and reporting. Agreater respect for the reasons behind these rules will help to ensure the objectiveness and(therefore) scientific character of contemporary clinical research.\Be Office ofHealth Technology AssessmentRules 01scienlilic inqUiry17References1 Green M. Perception, interpretation and the sciences. In: Depew D, Weber B, (editors).Evolution at a crossroads. Cambridge, MA: MIT Press; 1985.2 Longino HE. Science as social knowledge. Princeton: Princeton University Press; 1990.3 Merton, R. Some preliminaries to a sociology of medical education. In: R. Merton et al.,(editors). The Student-physician: Introductory studies in the sociology of medical education.Cambridge, MA: Harvard University Press; 1957.4 Kuhn T. Objectivity, values and theory choice. In: Butts R, Hintikka J, (editors). Historical andphilosophical dimensions of logic methodology, and philosophy of science. Dordrecht: R. Reidel;1977.5 Kosso P. Reading the book of nature: an introduction to the philosophy of science. Cambridge:Cambridge University Press; 1992.6 Rosenthal R. Projection, excitement and unconscious experimenter bias [abstract]. AmericanPsychologist 1958;13:345-346.7 Rosenthal R, Fode KL. The problem of experimenter outcome-bias. In: Ray DP, (editor). Seriesresearch in social psychology. Symposia studies series, no. 8. Washington, DC: National Instituteof Social and Behavioural Science; 1961.8 Rosenthal R. Experimenter effects in behavioral research. New York: Appleton-Century-Crofts;1966.9 Ross L, Anderson CA. Shortcomings in the attribution process: On the origins and maintenanceof erroneous social assessments. In: Kahneman D, Slovic P, Tversky A. Judgment underuncertainty: heuristics and biases. Cambridge: Cambridge University Press; 1982.Be Office ofHealth Technology AssessmentRules ofscienb"lic inquiry18,',10 Taylor SE, Fiske ST. Point of view and perceptions of causality. J Personality Soc Psychol1975;32:439-445.11 Jennings DL, Amabile TM, Ross L. Informal covariation assessment: data-based versus theory-based judgments. In: Kahneman D, Slovic P, Tversky A. Judgment under uncertainty: heuristicsand biases. Cambridge: Cambridge University Press; 1982.12 Brophy, J.E. Research on the self-fulfilling prophecy and teacher expectations. Journal ofEducational Psychology 1983;75:631·61.13 Jussim, L. Teacher expectations: Self-fulfilling prophecies, perceptual biases. Journal ofPersonality and Social Psychology 1989;57:469-80.14 Eden, D. Pygmalion in management: Productivity as a self-fulfilling prophecy. Lexington:Lexington BooksID.C. Heath and Company; 1990.15 Harris, M.J. Self-fulfilling prophecies in the clinical context: review and implications forclinical practice. Applied & Preventive Psychology 1994;3(3): 145-158.16 Jones DR. Bias and confounding in clinicalstudies. In: Nimmo W, Tucker G, editors. Clinicalmeasurement in drug evaluation. New York: John Wiley & Sons; 1995. p. 176.17 Fletcher RH. Fletcher SW, Wagner EH. Clinical epidemiology: The essentials. 3rd ed.Baltimore: Williams & Williams; 1996.18 Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality ofreporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276(8):637-9.19 Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical trials. 3rd ed. St. Louis:Mosby; 1996 :82-93.20 Jenicek M. Epidemiology: the logic of modem medicine. Montreal: Epimed; 1995.BeOffice ofHealth Technology AssessmentRules ofscientific inquiry1921 Chalmers TC, Celano P, Sacks HS, Smith H. Bias in treatment assignment in controlled clinicaltrials. N Engl J Med 1983;309:1358-1361.22 Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions ofmethodological quality associated with estimates of treatment effects in controlled trials. JAMA1995;273(5):408-12.23 Chalmers T. The challenges of clinical measurement. Mt Sinai J Med 1983;50:138-140.24 Gotzsche PC. Methodology and overt and hidden bias in reports of 196 double-blind trials ofnonsteroidal antiinflammatory drugs in rheumatoid arthritis. Controlled Clin Trials 1989;10(1):31-56.25 Gotzsche PC. Blinding during data analysis and writing of manuscripts. Controlled Clin Trials1996;17(4):285-90.26 Koran LM. The reliability of clinical methods, data and judgments. 1.N Eng/ J Med1975;293:642-646.27 Koran LM. The reliability of clinical methods, data and judgments. II. N Engl J Med1975;293:695-701.28 Feinstein AR. A bibliography of publications on observer variability. J Chron Dis1985;38:619-632.29 Elmore JG, Feinstein AR. A bibliography of publications on observer variability (Finalinstallment). J Clin Epidemiol 1992;45:567-580.. ., 30 Tomasello F, Mariani F, Fieschi C, Argentino C, Bono G, De Zache L, et al. Assessment of inter-observerdifferences in the Italian multicenter study on reversible cerebral ischemia. Stroke 1982;13:32-35.Be Office ofHealth Technology AssessmentRules ofscientific inquiry2031 Shinar D, Gross CR, Mohr JP, Caplan LR, Price TR, WolfPA et al. Interobservervariability in theassessment of neurologic history and examination in the Stroke Data Bank. Arch Neurol1985;42:557-65.32 Green CJ, Hadom DC, Bassett K, Kazanjian A. Anticoagulation in chronic nonvalvular atrialfibrillation: a critical appraisal and meta-analysis. Can J Cardioll997; 13(9):811-815.33 McConnack JP, Levine M. Meaningful interpretation of risk reduction from clinical drug trials.Ann Pharmacother 1993;27:1272-1277.34 Naylor DC, Chen E, Strauss B. Measured enthusiasm: does the method of reporting trial resultsalter perceptions of therapeutic effectiveness? Ann Intern Med 1992;117:916-921.35 Laupacis A, Sackett DL, Roberts RS. An assessment of clinically useful measures of theconsequences of treatment. N Engl J Med 1988;318:1728-1733.36 Laupacis A, Naylor CD, Sackett 01. How should the results of clinical trials be presented toclinicians? ACP Journal Club May/June 1992.37 Forrow L, Taylor WC, Arnold RM. Absolutely relative: how research results are summarizedcan affect treatment decisions. Am J Med 1992;92: 121-124.38 Atrial Fibrillation Investigators. Risk factors for stroke and efficacy of antithrombotic therapyin atrial fibrillation: analysis of pooled data from five randomized controlled trials. Arch InternMed 1994;154:1449-57.39 Green CJ, Hadom 0, Kazanjian A. Anticoagulation for stroke prevention in chronic non-valvular atrial fibrillation. Vancouver: British Columbia Office of Health TechnologyAssessment. 95:20; 1995.Be Office ofHealth Technology AssessmentRules ofscienfffic inquiry21


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items