UBC Faculty Research and Publications

Chest radiograph reading and recording system: evaluation in frontline clinicians in Zambia Henostroza, German; Harris, Jennifer B; Kancheya, Nzali; Nhandu, Venerandah; Besa, Stable; Musopole, Robert; Krüüner, Annika; Chileshe, Chisela; Dunn, Ian J; Reid, Stewart E Mar 23, 2016

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-12879_2016_Article_1460.pdf [ 403.61kB ]
JSON: 52383-1.0366829.json
JSON-LD: 52383-1.0366829-ld.json
RDF/XML (Pretty): 52383-1.0366829-rdf.xml
RDF/JSON: 52383-1.0366829-rdf.json
Turtle: 52383-1.0366829-turtle.txt
N-Triples: 52383-1.0366829-rdf-ntriples.txt
Original Record: 52383-1.0366829-source.json
Full Text

Full Text

RESEARCH ARTICLE Open AccessChest radiograph reading and recordingsystem: evaluation in frontline clinicians inZambiaGerman Henostroza1,2*, Jennifer B. Harris1,2,3, Nzali Kancheya2, Venerandah Nhandu2, Stable Besa2,Robert Musopole2, Annika Krüüner1,2, Chisela Chileshe4, Ian J. Dunn5 and Stewart E. Reid2,6AbstractBackground: In Zambia the vast majority of chest radiographs (CXR) are read by clinical officers who have limitedtraining and varied interpretation experience, meaning lower inter-rater reliability and limiting the usefulness of CXR asa diagnostic tool. In 2010–11, the Zambian Prison Service and Ministry of Health established TB and HIV screeningprograms in six prisons; screening included digital radiography for all participants. Using front-line clinicians weevaluated sensitivity, specificity and inter-rater agreement for digital CXR interpretation using the Chest RadiographReading and Recording System (CRRS).Methods: Digital radiographs were selected from HIV-infected and uninfected inmates who participated in aTB and HIV screening program at two Zambian prisons. Two medical officers (MOs) and two clinical officers(COs) independently interpreted all CXRs. We calculated sensitivity and specificity of CXR interpretationscompared to culture as the gold standard and evaluated inter-rater reliability using percent agreement andkappa coefficients.Results: 571 CXRs were included in analyses. Sensitivity of the interpretation “any abnormality” ranged from50–70 % depending on the reader and the patients’ HIV status. In general, MO’s had higher specificities thanCOs. Kappa coefficients for the ratings of “abnormalities consistent with TB” and “any abnormality” showedgood agreement between MOs on HIV-uninfected CXRs and moderate agreement on HIV-infected CXRswhereas the COs demonstrated fair agreement in both categories, regardless of HIV status.Conclusions: Sensitivity, specificity and inter-rater agreement varied substantially between readers withdifferent experience and training, however the medical officers who underwent formal CRRS training hadmore consistent interpretations.Keywords: chest radiograph, x-ray, Zambia, Chest Radiograph Reading and Recording System (CRRS)BackgroundDespite global progress in tuberculosis (TB) preventionand control, TB remains a leading cause of morbidityand mortality in sub-Saharan Africa, especially amongpersons with HIV [1]. Accurate diagnosis is a majorchallenge with sputum-smear microscopy and chestradiography (CXR) still the primary diagnostic tools inmany countries. Smear microscopy is less than 50 % sen-sitive in HIV-infected patients [2, 3], leaving many diag-noses reliant on CXR and clinical findings. In somesettings, CXR is also used as a screening tool to identifyTB suspects. Unfortunately CXR interpretation is com-plex and dependent on the skill of the reader and thequality of the x-ray.Digital radiography has increased optimism for theuse of CXRs as it offers consistent, better quality im-ages and lower running costs than analog radiology[4, 5]. Other efforts to improve CXR accuracy focuson standardizing interpretation. An example is the* Correspondence: germanh@uab.edu1Department of Medicine, Division of Infectious Diseases, University ofAlabama at Birmingham, Birmingham, USA2Centre for Infectious Disease Research in Zambia, Lusaka, ZambiaFull list of author information is available at the end of the article© 2016 Henostroza et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.Henostroza et al. BMC Infectious Diseases  (2016) 16:136 DOI 10.1186/s12879-016-1460-zChest X-Ray Reading and Recording System (CRRS) formdeveloped by the Lung Institute at the University of CapeTown [6, 7]. Experience with the CRRS to date hasbeen reported primarily for sub-specialist providers,most of whom were certified as CRRS readers.[8–11] This requires a five-day training course [9],available only in South Africa, potentially limitingaccess for individuals from other countries with fi-nancial constraints.In resource-limited settings (RLS) there is often ashortage of medical specialists resulting in CXR inter-pretation being performed by medical officers (MOs)and mid-level providers such as clinical officers (COs).With less training and experience in CXR interpretationthan specialists, they likely have lower inter-rater reli-ability which further limits the usefulness of CXR as adiagnostic tool [12]. To date there is no data on the per-formance of CRRS in general practitioners and mid-levelproviders.In Zambia the annual TB incidence is 427/100,000and 64 % of TB patients are HIV-infected [1]. Only29 % of HIV-infected pulmonary TB cases are smear-positive [13] and neither culture nor Xpert MTB/RIFare routinely available, so CXR plays an importantrole in TB diagnosis. However the vast majority ofCXRs are read by clinical officers who have limitedtraining and varied experience. We evaluated digitalCXR interpretation using the CRRS form (Version2007) in frontline clinicians in Zambia including twoMOs and two COs.MethodsIn 2010–11, the Zambian Prison Service and Ministry ofHealth established TB and HIV screening programs insix prisons with funding from the TB REACH initiativeof the Stop TB Partnership and technical support fromthe Centre for Infectious Disease Research in Zambia(CIDRZ). The overall goal of this program was to de-velop capacity to ensure that TB and HIV screeningwere conducted for all inmates entering and residing inthese facilities.TB and HIV screening protocolScreening procedures have been described elsewhere[14]. Inmates were assessed for self-reported TB symp-toms and other TB risk factors. Regardless of whetherthey had symptoms, all inmates submitted two sputa forflorescence microscopy (FM), had a digital CXR taken,and underwent physical examination. The project COmade an initial TB diagnosis based on history, physicalexam, CXR interpretation and FM smear results. Inaddition, inmates with an unknown HIV status were of-fered testing.Laboratory proceduresTwo sputa per inmate underwent FM. One sputum perinmate was cultured using one of two algorithms: (a)both liquid (BD BACTEC™ MGIT™ 960 MycobacteriaTesting System) and solid (BD BBL™ Lowenstein-JensenMedium) media or (b) two tubes of liquid media withthe manual Mycobacteria Growth Indicator Tube system(BD BBL™ MGIT™ Mycobacterial Growth IndicatorTube). M. tuberculosis complex (MTBC) speciation anddrug susceptibility was performed by GenotypeMTBDRplus (Hain Life Sciences, Germany) line probeassay.Case definitionA TB case was defined as culture positive with speciesidentified as MTBC.Chest X-ray selectionWe selected a sample of CXRs as follows: (1) all patientswith CXRs deemed abnormal by the project CO; (2) allpersons with normal CXRs who were diagnosed with TB(based on smear results, clinical criteria, and/or cultureconfirmation); and (3) a random sample of inmates withnormal CXRs and not diagnosed with TB. After evaluat-ing the number of CXRs in categories (1) and (2), we de-cided to select 80 CXRs from HIV-positive and 80 fromHIV-negative inmates in category (3) to strike a balancebetween feasibility to conduct all CXR readings and en-suring that there were an adequate number of normalCXRs in the sample. Our only exclusion criteria werecurrent or recent TB treatment or having an unknownHIV status.Chest X-ray interpretationTwo Zambian MOs with over ten years of experience indiagnosing and treating tuberculosis patients and twoCOs with more than 5 years of experience were invitedto participate in this evaluation. The MOs were Zambiangraduates from the University of Zambia School ofMedicine and the Zaparozhe Medical Institute, Ukraine;and the COs were local graduates of a 3-year program inclinical medicine, surgery and paediatrics. In addition,the two MOs attended the CRRS five-day trainingcourse in May 2010 in South Africa (using CRRS 2010guidelines) where they were certified as “B-grade”readers. “B grade readers” are defined as those who takethe course and pass the certification exam [15]. TheCOs did not attend the official CRRS training; insteadthey received a four-hour orientation provided by a non-CRRS trained radiologist who is a certified radiologist bythe Royal College of Physicians of Canada and holds afaculty position within the Radiology Department at theUniversity of British Columbia. The orientation con-sisted of a presentation and discussion of the CRRS formHenostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 2 of 8following the “Instructions for the use of the ChestReading and Recording System” (using CRRS 2010guidelines), followed by 20 h evaluating chest radio-graphs using the form.The CRRS uses a simplified and systematic approachto CXR reading and interpretation with readers complet-ing a form documenting CXR findings including the typeof abnormality present (parenchymal, large/small opaci-fications, cavitation, pleural and central abnormalities)(Additional file 1). Readers are then required to make afinal assessment whether the radiograph is “completelynormal” and if not, whether the abnormalities found are“consistent with TB.” CXRs were read using the RoganDelft View Pro-X (Version, Veenendal, NL) view-ing software. Computer stations had monitors with aresolution of 1280 × 720 pixels. All readers were blindedto clinical data except for HIV status and did not haveaccess to other readers’ reports.Data collection and analysisBecause we screened all inmates for TB, regardless ofpresenting characteristics, the vast majority were not di-agnosed with TB and had CXRs that were classified asnormal by the project CO. Thus we used disproportion-ate stratified sampling to maximize the number of ab-normal CXRs included in this study. We selected allCXRs deemed “abnormal” by the study CO as well as allCXRs from inmates diagnosed with TB. We selected asubset of CXRs from patients who were not diagnosedwith TB and were deemed to have “normal” CXRs bythe study CO.Data were collected using CRRS forms configured intoan electronic format using MS Access (Microsoft) andVisual Basic (Microsoft). Readers entered data directlyinto the electronic record. The system had features tominimize data entry errors including consistency checksand automatic skips. All data were exported into SAS9.3 (Cary, North Carolina, USA) for subsequent cleaningand analysis.We calculated sensitivity and specificity of the CXR in-terpretations “any abnormality” and “abnormalities con-sistent with TB” for each reader using TB culture as thegold standard. We assessed inter-rater reliability withpercent agreement and kappa coefficients between thetwo CRRS-certified MOs (Reader 1 & Reader 2) and thetwo CRRS-oriented COs (Reader 3 & Reader 4). Percentagreement and kappa coefficients were calculated for 8major abnormality classifications on the CRRS system:parenchymal abnormalities, large opacifications, smallopacifications, cavitation, pleural abnormalities, centralabnormalities, any abnormality and abnormalities con-sistent with TB. Ninety-five percent confidence intervalswere calculated for all measures. Kappa coefficients wereinterpreted as follows: ≤0.2 was considered pooragreement; 0.21–0.40 was fair; 0.41–0.60 was moderate;0.61–0.80 was good; and >0.80 was very good. Perform-ance measures were compared between clinician groups(MOs and COs) for obvious trends. Because each grouphad only two readers that were selected for convenience,we did not summarize measures within clinician groupsor conduct statistical tests to compare groups.Ethics statementThe protocol was approved by the Biomedical ResearchEthics Committee of the University of Zambia (001–03–11), the Zambian Ministry of Health and the Institu-tional Review Boards of the University of Alabama atBirmingham (F101014011) and the University of NorthCarolina at Chapel Hill, United States of America.A waiver of informed consent and documentation wasapproved by both the above named ethics committees.This was a retrospective analysis of de-identified elec-tronic data collected under a previously approved proto-col and stored using a unique identifying number,meaning it was not feasible to trace all participantsscreened within the program.ResultsBetween January and July 2011, 3405 inmates without acurrent or recent history of TB were screened for TBand HIV. 3160 agreed to HIV testing or had a knownprior status. From 711 HIV-positive inmates, 235 CXRswere selected as follows: 137/137 CXRs deemed abnor-mal by the project CO; 18/18 normal CXRs from pa-tients who were diagnosed with TB based on smear,culture, and/or clinical criteria; and 80/556 normalCXRs randomly selected from patients not diagnosedwith TB. From 2449 HIV-negative inmates, 339 CXRswere selected as follows: 236/236 CXRs deemed abnor-mal by the project CO; 23/23 normal CXRs from pa-tients who were diagnosed with TB based on smear,culture, or clinical criteria, and 80/2190 normal CXRsrandomly selected from patients not diagnosed with TB.Of the 574 images selected, three could not be inter-preted due to file corruption.Patients and case descriptionOf 571 patients included in analyses, 233 (41 %) wereHIV-infected. The inmates’ mean age was 38.6 years,97.4 % were male, and 73.6 % presented with at leastone TB symptom. One fourth of them (25.2 %) had aprior history of TB and 503 (88.1 %) had a valid cultureresult. Of these, 74 (14.7 %) had culture-confirmed TB;30/200 (15 %) among HIV-infected and 44/303 (14.5 %)among HIV-uninfected (Table 1).Henostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 3 of 8Chest X ray sensitivity and specificity compared to cultureSixty-eight participants (33 HIV-positive and 35 HIV-negative) had missing or contaminated cultures andwere excluded from sensitivity and specificity analyses.We first assessed sensitivity and specificity of the classi-fication “abnormalities consistent with TB”. Comparedto culture, the CRRS-certified MOs’ readings had sensi-tivities of 57 and 50 % and specificities of 61 and 60 % inCXRs from HIV-infected inmates. For CXRs from HIV-uninfected inmates, the MOs’ sensitivities were 61 and55 % with specificities of 70 and 55 %. The CRRS-oriented COs had sensitivities 67 and 53 % with specific-ities of 42 and 37 % in HIV-infected inmates. AmongHIV-uninfected patients, sensitivities for COs were 68and 61 % with specificities of 38 and 37 %. When webroadened the CXR classification to “any abnormalities,”point estimates for sensitivity increased slightly for twoof the four readers, but did not change for the other two(Table 2).With both classifications (“abnormalities consistentwith TB” and “any abnormalities”), three of the fourreaders had slightly higher sensitivities with CXRs fromHIV-negative persons than with CXRs from HIV-positive persons (Table 2), however all differences weresmall and confidence intervals overlapped substantially.Comparing the CRRS-certified MOs to the CRRS-oriented COs, the only consistent trend was higher spec-ificities for the MOs (Table 2).Inter rater reliabilityPercent agreementPercent agreement and kappa statistics are shown inTable 3. Percent agreement between CRRS-certifiedMOs on identification of specific abnormalities rangedfrom 73 to 87 % for HIV-infected patients and 76 to96 % for HIV-uninfected patients. For CRRS-orientedCOs, percent agreements ranged from 65 to 93 % inHIV-infected patients and 69 to 84 % in HIV-uninfectedpatients.Kappa coefficientThe MOs’ kappa coefficients for “abnormalities consist-ent with TB,” were 0.49 for HIV-positive and 0.70 forHIV-negative CXRs. Kappas for “any abnormality” were0.46 for HIV-positive and 0.62 for HIV-negative CXRs.Kappa coefficients for specific chest abnormalitiesTable 1 Cohort characteristicsCharacteristic HIV positive participants HIV negative participants All participantsn = 233 n = 338 n = 571Male sex 220 (94.4 %) 336 (99.4 %) 556 (97.4 %)Age, mean (SD) 38.1 (8.4) 39.0 (12.9) 38.6 (11.3)Prior history of TBa 83 (35.6 %) 61 (18.0 %) 144 (25.2 %)Culture-confirmed TBb 30/200 (15.0 %) 44/303 (14.5 %) 74/503 (14.7 %)Smear positive TB 9 (3.9 %) 14 (4.1 %) 23 (4.0 %)Any TB-related symptomsa 179 (76.8 %) 241 (71.3 %) 420 (73.6 %)aSelf-reported cough, fever, weight loss, night sweats, difficulty breathing, chest pain, loss of appetite, or swelling (lymphadenopathy)b68 patients excluded due to missing/contaminated culturesTable 2 Sensitivity & Specificity compared to culture, stratified by HIV statusHIV Positive HIV Negativen = 30 culture-confirmed TB; 170 no TBa n = 44 culture-confirmed TB; 259 no TBAbnormalities consistent with TB Any abnormality Abnormalities consistent with TB Any abnormalitySensitivity Specificity Sensitivity Specificity Sensitivity Specificity Sensitivity SpecificityCRRS-certified medical officersReader 1 0.57 0.61 0.63 0.50 0.61 0.59 0.70 0.47(0.37–0.75) (0.53–0.68) (0.44–0.80) (0.42–0.58) (0.45–0.76) (0.53–0.65) (0.54–0.83) (0.41–0.54)Reader 2 0.50 0.60 0.50 0.59 0.55 0.61 0.55 0.60(0.31–0.69) (0.52–0.67) (0.31–0.69) (0.51–0.66) (0.39–0.70) (0.55–0.67) (0.39–0.70) (0.54–0.66)CRRS-oriented clinical officersReader 3 0.67 0.42 0.77 0.38 0.68 0.38 0.70 0.33(0.47–0.83) (0.34–0.50) (0.58–0.90) (0.31–0.46) (0.52–0.81) (0.32–0.44) (0.55–0.83) (0.27–0.39)Reader 4 0.53 0.37 0.53 0.35 0.61 0.37 0.61 0.33(0.34–0.72) (0.30–0.45) (0.34–0.72) (0.28–0.43) (0.45–0.76) (0.32–0.44) (0.45–0.76) (0.27–0.39)a33 HIV–positive and 35 HIV–negative participants excluded due to missing or contaminated culturesHenostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 4 of 8Table 3 Inter–rater reliabilityAgreement Index ParenchymalabnormalitiesLargeopacificationsSmallopacificationsCavitation PleuralabnormalitiesCentralabnormalitiesAnyabnormalityAbnormalitiesconsistent withTBCRRS–certified medical officersHIV Positive patients (N = 231)aBoth readers agreeabnormality present38 8 24 2 32 17 78 69Only Reader 1 saysabnormality present56 38 36 30 13 21 41 28Only Reader 2 saysabnormality present7 5 10 0 16 17 22 29Both readers agreeabnormality not present130 180 161 199 170 176 90 105Percent Agreement(95 % CI)0.73 0.81 0.80 0.87 0.87 0.84 0.73 0.75(0.67–0.78) (0.76–0.86) (0.75–0.85) (0.83–0.91) (0.83–0.92) (0.79–0.88) (0.67–0.78) (0.70–0.81)Kappa (95 % CI) 0.38 0.20 0.40 0.10 0.61 0.38 0.46 0.49(0.27–0.50) (0.06–0.35) (0.26–0.53) (–0.03–0.24) (0.48–0.74) (0.22–0.54) (0.34–0.57) (0.38–0.61)Strength of agreement(based on kappa)Fair Poor Fair Poor Good Fair Moderate ModerateHIV Negative patients (N = 335)aBoth readers agreeabnormality present58 10 41 6 60 18 136 122Only Reader 1 saysabnormality present100 74 64 39 24 32 55 31Only Reader 2 saysabnormality present5 1 12 0 17 15 9 19Both readers agreeabnormality not present172 250 218 290 234 270 135 163Percent Agreement(95 % CI)0.96 0.78 0.77 0.88 0.88 0.86 0.76 0.85(0.64–0.74) (0.73–0.82) (0.73–0.82) (0.85–0.92) (0.84–0.91) (0.82–0.90) (0.72–0.81) (0.81–0.89)Kappa (95 % CI) 0.35 0.16 0.39 0.21 0.66 0.36 0.62 0.70(0.27–0.43) (0.07–0.26) (0.29–0.50) (0.07–0.35) (0.57–0.76) (0.21–0.50) (0.54–0.70) (0.62–0.77)Strength of agreement(based on kappa)Fair Poor Fair Fair Good Fair Good GoodCRRS-oriented clinical officersHIV Positive patients (N = 231)aBoth readers agreeabnormality present46 33 5 1 55 14 107 102Only Reader 3 saysabnormality present33 28 22 5 16 41 40 35Only Reader 4 saysabnormality present44 37 11 11 63 21 41 43Both readers agreeabnormality not present108 133 193 214 97 155 43 51Percent Agreement 0.67 0.72 0.86 0.93 0.66 0.73 0.65 0.66(0.61–0.73) (0.66–0.78) (0.81–0.90) (0.90–0.96) (0.60–0.72) (0.67–0.79) (0.59–0.71) (0.60–0.72)Kappa (95 % CI) 0.28 0.31 0.16 0.08 0.32 0.15 0.24 0.29(0.16–0.41) (0.18–0.44) (–0.02–0.34) (–0.12–0.28) (0.21–0.43) (0.01–0.29) (0.11–0.37) (0.17–0.42)Strength of agreement(based on kappa)Fair Fair Poor Poor Fair Poor Fair FairHenostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 5 of 8(cavities, opacifications and pleural, central or parenchy-mal abnormalities) ranged from 0.35 to 0.61 amongHIV-infected and from 0.29 to 0.66 among HIV-uninfected inmates.The COs had kappa coefficients for “abnormalitiesconsistent with TB” of 0.29 with HIV-infected and 0.36with HIV-uninfected CXRs. For “any abnormality” theCOs had kappas of 0.24 in HIV-positive and 0.32 inHIV-negative CXRs. Kappas for specific chest abnormal-ities ranged from 0.08 to 0.32 among HIV-infected andfrom 0.04 to 0.51 among HIV-uninfected.DiscussionThe value of CXR for TB screening and diagnosis hasshown wide variability in performance across differentsettings and patient populations [3, 9, 16, 17]. We evalu-ated the performance of digital radiography when inter-preted by front-line clinicians using CRRS forms inZambia. We provided COs with a local orientation tothe CRRS form to assess their performance with theunderlying rationale that the five-day CRRS trainingcourse in South Africa is not easily accessible to COs inZambia who perform much of the CXR interpretationfor TB diagnosis.Despite using digital radiographs, the sensitivity of theinterpretation “any abnormality” ranged from only 50–70 % depending on the reader and the patients’ HIV sta-tus. Thus if CXR was used as the sole TB screening tool inthis cohort, 30 to 50 % of the culture-confirmed TB caseswould have been missed. Even more cases may have beenmissed if the rating “abnormalities consistent with TB”was used, as sensitivities were slightly lower for two of thefour readers. The use of CXR abnormalities as diagnosticcriteria may result in over-diagnosis of cases since speci-ficities for the classification “abnormalities consistent withTB” ranged from 37–61 %. When comparing the CRRS-trained MOs to the CRRS-oriented COs, the MOs con-sistently had higher specificities. In contrast, there wereno consistent trends seen with sensitivities. Furthermore,sensitivities were not strongly influenced by HIV status.We looked at inter-rater reliability using kappa statis-tics. For the ratings of “abnormalities consistent withTB” and “any abnormality” the MO’s had “good” agree-ment for HIV-uninfected CXRs but only “moderate”agreement for HIV-infected CXRs. This is consistentwith literature showing that HIV-infected TB patientspresent with broad array of atypical radiological abnor-malities [18–21]. When looking at specific types of ab-normalities, the agreement between MOs was verysimilar for HIV-infected and HIV-uninfected patients,but their agreement was “good” only for pleural abnor-malities; the rest were in the “poor” or “fair” ranges. Thissuggests that MOs had better agreement on overall in-terpretation of CXRs than on specific abnormalities.The CRRS-oriented COs demonstrated “poor” or “fair”agreement for almost all categories. Unlike the MOs, theydid not have better agreement on the overall assessmentcategories than they did for the specific abnormalities.Looking at specific abnormalities, they achieved “moder-ate” agreement for parenchymal abnormalities in HIV-negative patients and “fair” agreement for large opacitiesand pleural abnormalities. This agreement in identifyinggross, more easily observable radiographic abnormalitiesmight be expected given their level of training. Some ofthe lowest kappas for both MOs and COs were observedwith cavities, which is a concern given the high correlationof this abnormality with pulmonary TB. Other studieshave similar findings, even with expert readers [9].Readers were not blinded for HIV status to reflect actualcase scenario when evaluating TB suspects. Due to theTable 3 Inter–rater reliability (Continued)HIV Negative patients (N = 335)aBoth readers agreeabnormality present88 58 9 1 107 36 177 165Only Reader 3 saysabnormality present42 38 43 10 18 70 52 51Only Reader 4 saysabnormality present36 48 11 15 86 22 48 48Both readers agreeabnormality not present169 191 272 309 124 207 58 71Percent Agreement(95 % CI)0.77 0.74 0.84 0.92 0.69 0.73 0.70 0.70(0.72–0.81) (0.70–0.79) (0.80–0.88) (0.90–0.95) (0.64–0.74) (0.68–0.77) (0.65–0.75) (0.66–0.75)Kappa (95 % CI) 0.51 0.39 0.18 0.04 0.40 0.28 0.32 0.36(0.41–0.60) (0.28–0.50) (0.05–0.31) (–0.10–0.18) (0.31–0.49) (0.17–0.38) (.21–0.42) (0.25–0.46)Strength of agreement(based on kappa)Moderate Fair Poor Poor Fair Fair Fair Faira2 CXRs among HIV-positive and 3 among HIV-negative patients were deemed ‘Unreadable’ by one or more reader and excluded from analysesHenostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 6 of 8Zambian opt out approach to HIV testing most TB sus-pects will have an HIV test result that can be accessed bythe radiologist.The explanation for differences in performance of sen-sitivity and specificity between MOs and COs is likelymultifactorial but likely most related to the level ofhealth care worker training. With only a three-yeartraining program, the COs had less clinical instruction,training, and mentorship. In addition, our study COs didnot complete the formal CRRS training. Our evaluationwas based on an active case finding intervention in ahigh risk population. Therefore, other pathologies couldbe responsible for abnormal radiological findings in con-junction with clinicians with a high index of suspicionfor TB.The low kappa statistics among COs suggests that ashort orientation to the CRRS form is not sufficient todevelop acceptable CXR interpretation skills for peoplewith their level of training. Since the formal CRRS train-ing is inaccessible to most frontline clinicians outside ofSouth Africa, alternatives could include a “Trainer ofTrainers” curriculum such that CRRS-certified “trainers”could return to their respective sites to train and mentorothers. Ongoing mentoring to address skills deficits offrontline providers might also be accomplished by an e-learning curriculum that provides reminders/refreshersusing text and images demonstrating variations of radio-graphic abnormalities [22]. When designing x-ray train-ing packages for COs, special attention should be placedon x-ray interpretation in HIV-infected patients and rec-ognition of cavities, given its high correlation with pul-monary tuberculosis [23]. A more simple classificationfor CXR interpretation could improve sensitivity andinter-rater reliability between clinical officers [24].These results highlight some of the challenges of usingCXR as a primary screening and/or diagnostic tool fornon-radiologists. Sensitivity, specificity and inter-rateragreement varied substantially based on reader experi-ence and training. The sensitivity and specificity of CXR,as well as the training of the health care providers whowill be interpreting CXRs, should be carefully consideredwhen implementing chest radiography in screening anddiagnostic algorithms. In addition, these findings suggestthat prison environments warrant a high index of suspi-cion for TB even among inmates with normal CXRssince the sensitivity of CXR was fairly low, regardless ofwho was reading the CXR.Strengths and limitationsThis evaluation had several strengths: the electronicCRRS form limited data transcription errors and thehigh quality of digital CXRs should have minimized in-consistencies due to poor quality radiographs. Otherstrengths are the inclusion of both HIV-infected anduninfected, symptomatic and asymptomatic patientswho had culture results to serve as a gold standard forTB diagnostic status. This provided a diverse studypopulation in which to evaluate inter-observer agree-ment and diagnostic performance.We also had a few limitations. First of all, we had onlytwo MOs and two COs so it is difficult to make general-izations about classes of frontline providers, especiallysince the MOs had received CRRS training but the COshad not. Another limitation is that our radiologist wasnot CRRS trained, this could have affected the correctinterpretation of the CRRS form as was intended to beused. In addition, we only cultured one sputum per per-son which could have resulted in a few patients whotruly had TB being classified as “TB negative.” If thishappened, it could have resulted in slightly lower speci-ficities for CXR interpretation. However, all personswere cultured with both liquid and solid culture and wedo not believe this would have had a substantial effecton results. Because we screened all inmates, regardlessof presenting characteristics, many TB patients wereprobably caught at an early stage of disease. As such,their CXRs may not be typical of patients with more ad-vanced TB disease. Also due to our screening setting, weelected to use a non-random sample of CXRs to ensurethat the selection included culture-confirmed cases (forsensitivity assessment) as well as abnormal CXRs for as-sessment of agreement. However the COs and MOswere blinded to the CXR selection process, thus thisshould not have subjectively affected their assessments.Finally, Kappa statistics are affected by the prevalence ofthe assessed condition [25] and thus should be inter-preted with caution. A study in Kenya found lowerkappas when assessing CXRs from persons without TBthan CXRs from TB patients [24]. This suggests thatkappas in our study may have been even lower had therebeen a higher proportion of CXRs from persons withoutTB.ConclusionsThe WHO’s 2013 guidelines for systematic TB screen-ing recommend use of chest radiography for TBscreening as it is more sensitive than symptom-basedalgorithms [26]. Our study suggests that this approachmay not be highly sensitive in some settings and maybe limited by poor consistency in CXR interpretationamong frontline providers. The CRRS system createsa structure for CXR interpretation which should behelpful for novice or less experienced readers, how-ever we found that the CRRS form alone is not asubstitute for (a) formal training/experience in identi-fying specific CXR abnormalities (b) knowledge/ex-perience in deciding which abnormalities are likely tobe caused by TB. However in settings with fewHenostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 7 of 8radiologists, a tool such as CRRS might represent aviable option if combined with onsite trainers, men-toring and constant feedback.Additional fileAdditional file 1: Chest Radiograph Reading and Recording System.(PDF 247 kb)AbbreviationsCIDRZ: Centre for Infectious Disease Research in Zambia; COs: Clinicalofficers; CRRS: Chest Radiograph Reading and Recording System; CXR: Chestradiograph; FM: Florescence microscopy; MOs: Medical officers; MTBC: M.tuberculosis complex; RLS: Resource-limited settings; TB: Tuberculosis.Competing interestsThe authors declare that they have no competing interests.Authors’ contributionsGH, NK, JBH, IJD and SER developed the initial concept of the study andrevised subsequent manuscripts. GH, JBH and SR developed the analyticalplan. GH, JBH, and SR analyzed the data. NK, VN, SB and RM interpreted thechest x-rays. All authors read and approved the final manuscript. GH ascorresponding author had full access to all the data in the study and hadfinal responsibility for the decision to submit for publication. The authorsdeclare that they have no competing interests.AcknowledgementsWe would like to thank the Zambian Ministry of Home Affairs for theirsupport to this study. Funding for the Zambian Prison TB screening project’simplementation and evaluation was provided by the TB REACH Initiative ofthe Stop TB Partnership (T9-370-114ZAM). There was no additional fundingfor this secondary analysis.Author details1Department of Medicine, Division of Infectious Diseases, University ofAlabama at Birmingham, Birmingham, USA. 2Centre for Infectious DiseaseResearch in Zambia, Lusaka, Zambia. 3Department of Epidemiology,University of Alabama at Birmingham, Birmingham, USA. 4Prisons HealthServices, Ministry of Home Affairs, Lusaka, Zambia. 5Department of Radiology,University of British Columbia, Vancouver, Canada. 6Department of Medicine,Institute of Global Health and Infectious Diseases, University of NorthCarolina at Chapel Hill, Chapel Hill, USA.Received: 25 February 2015 Accepted: 10 March 2016References1. WHO. WHO Global Tuberculosis Report 2013. Geneva: World HealthOrganization; 2013. Contract No.: WHO/HTM/TB/2013.11.2. Nguyen DT, Nguyen HQ, Beasley RP, Ford CE, Hwang LY, Graviss EA.Performance of Clinical Algorithms for Smear-Negative Tuberculosis in HIV-Infected Persons in Ho Chi Minh City, Vietnam. Tuberc Res Treat. 2012;2012:360852. Pubmed Central PMCID: 3512270, Epub 2012/12/12. eng.3. Cain KP, McCarthy KD, Heilig CM, Monkongdee P, Tasaneeyapan T, KanaraN, et al. An algorithm for tuberculosis screening and diagnosis in peoplewith HIV. N Engl J Med. 2010;362(8):707–16. Epub 2010/02/26. eng.4. Story A, Aldridge RW, Abubakar I, Stagg HR, Lipman M, Watson JM, et al.Active case finding for pulmonary tuberculosis using mobile digital chestradiography: an observational study. Int J Tuberc Lung Dis. 2012;16(11):1461–7. Epub 2012/09/18. eng.5. Iademarco MF, O'Grady J, Lonnroth K. Chest radiography for tuberculosisscreening is back on the agenda. Int J Tuberc Lung Dis. 2012;16(11):1421–2.Epub 2012/10/10. eng.6. Den Boon S, Bateman ED, Enarson DA, Borgdorff MW, Verver S, Lombard CJ,et al. Development and evaluation of a new chest radiograph reading andrecording system for epidemiological surveys of tuberculosis and lungdisease. Int J Tuberc Lung Dis. 2005;9(10):1088–96. Epub 2005/10/19. eng.7. den Boon S, White NW, van Lill SW, Borgdorff MW, Verver S, Lombard CJ,et al. An evaluation of symptom and chest radiographic screening intuberculosis prevalence surveys. Int J Tuberc Lung Dis. 2006;10(8):876–82.Epub 2006/08/11. eng.8. Agizew T, Bachhuber MA, Nyirenda S, Makwaruzi VZ, Tedla Z, Tallaksen RJ,et al. Association of chest radiographic abnormalities with tuberculosisdisease in asymptomatic HIV-infected adults. Int J Tuberc Lung Dis. 2010;14(3):324–31.9. Dawson R, Masuka P, Edwards DJ, Bateman ED, Bekker LG, Wood R, et al.Chest radiograph reading and recording system: evaluation for tuberculosisscreening in patients with advanced HIV. Int J Tuberc Lung Dis. 2010;14(1):52–8. Pubmed Central PMCID: 3647461, Epub 2009/12/17. eng.10. Pinto LM, Dheda K, Theron G, Allwood B, Calligaro G, van Zyl-Smit R, et al.Development of a simple reliable radiographic scoring system to aid thediagnosis of pulmonary tuberculosis. PLoS One. 2013;8(1):e54235. PubmedCentral PMCID: 3548832.11. Pinto LM, Pai M, Dheda K, Schwartzman K, Menzies D, Steingart KR. Scoringsystems using chest radiographic features for the diagnosis of pulmonarytuberculosis in adults: a systematic review. Eur Respir J. 2013;42(2):480–94.12. Balabanova Y, Coker R, Fedorin I, Zakharova S, Plavinskij S, Krukov N, et al.Variability in interpretation of chest radiographs among Russian cliniciansand implications for screening programmes: observational study. BMJ. 2005;331(7513):379–82. Pubmed Central PMCID: 1184248, Epub 2005/08/13. eng.13. Team LDHM. In: Health ZMo, editor. Tuberculosis Report 2011. Lusaka:Team LDHM; 2011.14. Henostroza G, Topp SM, Hatwiinda S, Maggard KR, Phiri W, Harris JB, etal. The HIgh Burden of Tuberculosis (TB) and Human ImmunodeficiencyVirus (HIV) in a large Zambian prisson: a public health alert. PLoS One.2013;8(8):e67338.15. Institute UoCTL. Chest Radiogrpahic Reading and Recording System (CRRS)http://www.lunginstitute.co.za/content/talks.html 2013.16. van Cleeff MR, Kivihya-Ndugga LE, Meme H, Odhiambo JA, Klatser PR. Therole and performance of chest X-ray for the diagnosis of tuberculosis: acost-effectiveness analysis in Nairobi, Kenya. BMC Infect Dis. 2005;5:111.Pubmed Central PMCID: 1326228, Epub 2005/12/14. eng.17. van't Hoog AH, Meme HK, Laserson KF, Agaya JA, Muchiri BG, Githui WA, etal. Screening strategies for tuberculosis prevalence surveys: the value ofchest radiography and symptoms. PLoS One. 2012;7(7):e38691. PubmedCentral PMCID: 3391193, Epub 2012/07/14. eng.18. Aderaye G, Bruchfeld J, Assefa G, Feleke D, Kallenius G, Baat M, et al. Therelationship between disease pattern and disease burden by chestradiography, M. tuberculosis Load, and HIV status in patients with pulmonarytuberculosis in Addis Ababa. Infection. 2004;32(6):333–8. Epub 2004/12/15. eng.19. Johnson JL, Vjecha MJ, Okwera A, Hatanga E, Byekwaso F, Wolski K, et al.Impact of human immunodeficiency virus type-1 infection on the initialbacteriologic and radiographic manifestations of pulmonary tuberculosis inUganda. Makerere University-Case Western Reserve University ResearchCollaboration. Int J Tuberc Lung Dis. 1998;2(5):397–404. Epub 1998/06/05. eng.20. Noronha D, Pallangyo KJ, Ndosi BN, Lweno H, Sabuka SR. Radiological featuresof pulmonary tuberculosis in patients infected with human immunodeficiencyvirus. East Afr Med J. 1991;68(3):210–5. Epub 1991/03/01. eng.21. Asimos AW, Ehrhardt J. Radiographic presentation of pulmonarytuberculosis in severely immunosuppressed HIV-seropositive patients. Am JEmerg Med. 1996;14(4):359–63. Epub 1996/07/01. eng.22. Waitt CJ, Joekes EC, Jesudason N, Waitt PI, Goodson P, Likumbo G, et al. Theeffect of a tuberculosis chest X-ray image reference set on non-expert readerperformance. Eur Radiol. 2013;23(9):2459–68. Pubmed Central PMCID: 3738845.23. Van Dyck P, Vanhoenacker FM, Van den Brande P, De Schepper AM. Imaging ofpulmonary tuberculosis. Eur Radiol. 2003;13(8):1771–85. Epub 2003/08/28. eng.24. Hoog AH, Meme HK, van Deutekom H, Mithika AM, Olunga C, Onyino F,et al. High sensitivity of chest radiograph reading by clinical officers in atuberculosis prevalence survey. Int J Tuberc Lung Dis. 2011;15(10):1308–14.25. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problemof two paradoxes. J Clin Epidemiology. 1990;43(6):543–9.26. WHO. Systematic screening for active tuberculosis: Principles andrecommendations. Geneva, Switzerland: World Health Organization;2013. Contract No.: WHO/HTM/TB/2013.04.Henostroza et al. BMC Infectious Diseases  (2016) 16:136 Page 8 of 8


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items