UBC Faculty Research and Publications

A validation study of a clinical prediction rule for screening asymptomatic chlamydia and gonorrhoea… Falasinnu, Titilola; Gilbert, Mark; Gustafson, Paul; Shoveller, Jean 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Geographical validation.pdf [ 317.7kB ]
JSON: 52383-1.0228251.json
JSON-LD: 52383-1.0228251-ld.json
RDF/XML (Pretty): 52383-1.0228251-rdf.xml
RDF/JSON: 52383-1.0228251-rdf.json
Turtle: 52383-1.0228251-turtle.txt
N-Triples: 52383-1.0228251-rdf-ntriples.txt
Original Record: 52383-1.0228251-source.json
Full Text

Full Text

This version of the article “Falasinnu, T., Gilbert, M., Gustafson, P., & Shoveller, J. (2015). A validation study of a clinical prediction rule for screening asymptomatic chlamydia and gonorrhoea infections among heterosexuals in British Columbia. Sexually Transmitted Infections. doi: 10.1136/sextrans-2014-051992. Epub ahead of print.” is not the final version.   We acknowledge the publisher’s (BMJ) copyright.  Here is a link to the final published version of the article: http://www.ncbi.nlm.nih.gov/pubmed/25933609 ORIGINAL ARTICLEA validation study of a clinical prediction rule forscreening asymptomatic chlamydia and gonorrhoeainfections among heterosexuals in British ColumbiaQ1Q2Titilola FalasinnuQ3 ,1 Mark Gilbert,2 Paul Gustafson,3 Jean Shoveller1▸ Additional material ispublished online only. To viewplease visit the journal online(http://dx.doi.org/10.1136/sextrans-2014-051992).1The School of Population andPublic Health, University ofBritish Columbia, Vancouver,British Columbia, CanadaQ42British Columbia Center forDisease Control, Vancouver,British Columbia, Canada3The Department of Statistics,University of British Columbia,Vancouver, British Columbia,CanadaCorrespondence toDr Titilola Falasinnu, TheSchool of Population andPublic Health, University ofBritish Columbia, 2206 EastMall, Vancouver, BritishColumbia, Canada BC V6T1Z3; lola.falasinnu@ubc.caReceived 23 December 2014Revised 25 March 2015Accepted 11 April 2015To cite: Falasinnu T,Gilbert M, Gustafson P,et al. Sex Transm InfectPublished Online First:[please include Day MonthYear] doi:10.1136/sextrans-2014-051992ABSTRACTBackground One component of effective sexuallytransmitted infections (STIs) control is ensuring those athighest risk of STIs have access to clinical servicesbecause terminating transmission in this group willprevent most future cases. Here, we describe the resultsof a validation study of a clinical prediction rule foridentifying individuals at increased risk for chlamydia andgonorrhoea infection derived in Vancouver, BritishColumbia (BC), against a population of asymptomaticpatients attending sexual health clinics in othergeographical settings in BC.Methods We examined electronic records (2000–2012) from clinic visits at seven sexual health clinics ingeographical locations outside Vancouver. The model’scalibration and discrimination were examined by the areaunder the receiver operating characteristic curve (AUC)and the Hosmer–Lemeshow (H-L) statistic, respectively.We also examined the sensitivity and proportion ofpatients that would need to be screened at differentcut-offs of the risk score.Results The prevalence of infection was 5.3%(n=10 425) in the geographical validation population.The prediction rule showed good performance in thispopulation (AUC, 0.69; H-L p=0.26). Possible risk scoresranged from −2 to 27. We identified a risk score cut-offpoint of ≥8 that detected cases with a sensitivity of86% by screening 63% of the geographical validationpopulation.Conclusions The prediction rule showed goodgeneralisability in STI clinics outside of Vancouver withimproved discriminative performance compared withtemporal validation. The prediction rule has the potentialfor augmenting triaging services in STI clinics andenhancing targeted testing in population-basedscreening programmes.Q6Q7BACKGROUNDThe imperative to provide more efficient sexualhealth services by public health programmes hasled to the development of service models that opti-mise the use of health human resources, such asinternet-based sexually transmitted infections (STIs)testing and triage services.1–4 The aim of optimis-ing service provision could be facilitated by the useof risk estimation algorithms. In a previous paper,5a risk estimation algorithm for optimising asymp-tomatic chlamydia and gonorrhoea case finding wasderived using electronic medical records of patientvisits at two sexual health clinics in Vancouver,British Columbia (BC). This algorithm combinesfive risk factors: younger age, non-white race/ethni-city, multiple sexual partners, previous chlamydiadiagnosis and previous gonorrhoea diagnosis. Theprediction rule will eventually be adapted into atool for facilitating selective screening in GetChecked Online (GCO), a novel internet-basedtesting programme in BC.4 6In the derivation stage, we specified that themain intended application of the risk score is tohelp deal with the increasing numbers of peopleaccessing services in the sexual health clinic con-texts, particularly the asymptomatic individuals.However, a prediction rule’s accuracy in onecontext may vary from the performance estimatesreported in the derivation stage.7 8 Inconsistentperformance may reflect artifactual disparities(eg, different study contexts), potentially in com-bination with genuine disparities (eg, distributionof risk factors).9 Thus, it is important to considerthe prediction rule’s accuracy in varied settings,such as general practice clinics or hospitals, differ-ent types of hospitals or the same type of clinicalsetting in different geographical locations.9Before making recommendations regarding thewidespread use of the risk estimation algorithmderived through our previous work,5 we conductedan external validation study in an independentsetting to further test the parameters of the tool.It is essential to confirm that the algorithm is gener-alisable to a plausibly related setting (in addition tothe previous comparisons conducted with the der-ivation population) that reflects the level of hetero-geneity that will be encountered in real-lifeapplications of the algorithm.10 As was demon-strated previously, the algorithm showed reasonablediscrimination and calibration upon validation in twodifferent time periods (ie, temporal validation).11While temporal validation is often cited as the firststep in demonstrating the transferability of a predic-tion algorithm,12 it cannot assess the utility of thealgorithm to other clinics or cities.10 Geographical val-idation provides a more rigorous proof of validationthan temporal validation owing to the hypothesiseddifferences in patient mix, risk factor definitions anddisease prevalence.10It should be appreciated that a prediction rule’sperformance is often lower upon external valid-ation.13 In these settings, decision-makers have theoption to adopt a previously derived predictionrule if it is found to perform adequately or theycan derive or remodel a new prediction rule usingFalasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992 1Clinical123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128Please do not annotate this PDF with corrections - use the unmarked copy providedtheir own population. Here, we describe the results of a test toassess the generalisability of an algorithm derived in Vancouveragainst a population of patients attending sexual health clinicsin seven other geographical settings in BC. In addition, weexamine the implications of deriving a new prediction ruleusing the data from the geographical validation population(ie, remodelled prediction rule) and compare its performance tothe ‘Vancouver’ prediction rule.METHODSThe geographical validation dataset was derived from electronicmedical records of clients attending publicly funded sexualhealth clinics located in seven locations in BC between 2000and 2012: Penticton, Kelowna, Kamloops, New Westminster,Boundary, Courtenay and Prince George (see map in online sup-plementary figure 1). This analysis was limited to clinic visitsamong asymptomatic women and heterosexual men who arenot sexual contacts of STI cases and not receiving confirmatorypositive testing. The aim was to estimate the risk of chlamydiaand/or gonorrhoea infection. In the current paper, the originalmodel, regression coefficients and the simplified risk scoresderived from the Vancouver clinic data are applied to a geo-graphical validation population.10 A detailed description of thestudy protocol, analysis plan and predictor definitions has beenpreviously published.14The ‘Vancouver’ risk estimation algorithm uses a logisticregression formula to relate its five predictors to chlamydia orgonorrhoea risk. The regression coefficients and their associatedscoring points are listed in table 1. Multiple imputationmethods (five rounds) were used to impute missing values in thegeographical validation population.15–17 This analysis imputedmissing values using IVEware, a software application that per-forms multiple imputations of missing values using theSequential Regression Imputation Method.17 All predictors andthe outcome variables were included in the imputation modeland the results of the five imputed datasets were combined toobtain final estimates.15 16 To assess the performance of themodel in the geographical validation population, the model’sdiscrimination was estimated by calculating the area under thereceiver operating characteristic curve (AUC). The AUC givesthe likelihood that a randomly selected infected individualwould have a higher model predicted probability of chlamydiaor gonorrhoea infection than a randomly selected non-infectedindividual.18 The closer an AUC is to 100%, the better themodel.18Calibration was assessed with the Hosmer–Lemeshowgoodness-of-fit statistic, which investigates (under the nullhypothesis that there is no difference) the difference betweenthe model predictions and the actual observations using decilesof predicted probabilities to categorise patients.18 A p value>0.05 indicates a good fit.18 The model’s calibration was alsoexamined by graphically plotting the prevalence of chlamydiaand/or gonorrhoea infection in groups of the simplified riskscores. To aid population-based screening decision-making, arisk score was derived for each clinic visit in the geographicalvalidation population by adding up the scoring points derivedfrom table 1. An evaluation of the sensitivity (or the fraction ofinfected cases identified) and the proportion of the populationthat would be screened at different risk score cut points wasalso performed. A well-performing screening tool detects >90%of cases, while screening <60% of the population.19RESULTSDuring the years 2000–2012, there were 10 425 patient visitsthat met the inclusion criteria at sexual health clinics at the fol-lowing geographical sites: Penticton, Kelowna, Kamloops, NewWestminster, Boundary, Courtenay and Prince George. Onlinesupplementary figure 2 is a flow chart showing the selection ofclinic visits whose data comprised this validation study. Theprevalence of chlamydia and/or gonorrhoea infection was 5.3%(higher than the derivation population). Table 2 shows thedistribution of the baseline characteristics of patient visits.Table 1 Prediction rule for quantifying the probability of asymptomatic chlamydia and/or gonorrhoea infection among heterosexuals inVancouver, British ColumbiaVariable aOR (95% CI) Beta Scoring pointsIntercept – −6.0381 –Age (years)14–19 5.25 (2.82 to 9.79) 1.6589 820–24 1.92 (1.19 to 3.10) 0.6544 325–29 1.20 (0.74 to 1.94) 0.1794 130–39 0.71 (0.42 to 1.18) −0.3471 −2≥40 Ref Ref RefRace/ethnicityWhite Ref Ref RefNon-white 2.74 (2.03 to 3.70) 1.0093 5No. of sexual partners in previous six months0 Ref Ref Ref1–2 2.90 (0.97 to 8.75) 1.0565 5≥3 3.03 (0.99 to 9.30) 1.1000 6Previous chlamydia diagnosisYes 4.34 (3.19 to 5.89) 1.4574 7No Ref Ref RefPrevious gonorrhoea diagnosisYes 1.27 (0.70 to 2.34) 0.2426 1No Ref Ref RefaOR, adjusted Q10OR.2 Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992Clinical129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256Please do not annotate this PDF with corrections - use the unmarked copy providedThe derivation population (Vancouver) is included for compari-son. The majority of patient visits in the geographical validationpopulation had the following demographic characteristics: malegender (57.5%), aged between 20 and 24 years (28.0%), andwhite ethnicity (74.3%). More than two-thirds of patient visitsreported having 1–2 sexual partners in the previous six monthsand approximately 43% reported consistent condom use.Approximately 3% of patient visits reported injection drug useand the same proportion reported having sex with partnersrecruited online. Previous chlamydia diagnosis was reportedamong nearly 16% of patient visits.The geographical validation population differed from the der-ivation population by having a higher proportion of thefollowing characteristics: women, younger individuals, inconsist-ent condom use and injection drug use (table 2). There werealso some differences between the derivation and geographicalvalidation populations in terms of the unadjusted ORs examin-ing the associations between the predictors and the outcome.Gender and condom use were significantly associated with infec-tion in the geographical validation population—associations thatwere not significant in the derivation population (table 3). Race/ethnicity was not significantly associated with the outcome inthe geographical validation population unlike the derivationpopulation (table 3).The Vancouver risk model demonstrated good discrimination inthe geographical validation population. The AUC in theTable 2 Population characteristics of heterosexual visits at sexual health clinics in the derivation and geographical validation populationsDerivation population, 2000–2006Geographical validationpopulation, 2000–2012Variable N % N %Chlamydia or gonorrhoea caseYes 184 1.8 556 5.3No 10 253 98.2 9869 94.7GenderFemale 3496 33.5 4431 42.5Male 6941 66.5 5994 57.5Age (years)14–19 257 2.5 1748 16.820–24 1962 18.8 2922 28.025–29 2651 25.4 1969 18.930–39 3181 30.5 1850 17.7≥40 2386 22.9 1935 18.6Race/ethnicityWhite 7732 74.1 7741 74.3Non-white 2705 25.9 2684 25.7No. of sexual partners in previous six months0 644 6.2 590 5.71–2 6857 65.7 7004 67.2≥3 2936 28.1 2830 27.1Condom useNever 2362 22.6 2650 25.4Always 5269 50.5 4489 43.1Sometimes 2806 26.9 3286 31.5Sex with partners recruited onlineYes 417 4.0 341 3.3No 10 020 96.0 10 083 96.7Injection drug useYes 211 2.0 351 3.4No 10 226 98.0 10 073 96.6Sex with injection drug userYes 455 4.4 407 3.9No 9983 95.6 10 018 96.1Sex with commercial sex workerYes 1381 13.2 826 7.9No 9057 86.8 9599 92.1Previous chlamydia diagnosisYes 1518 14.5 1660 15.9No 8919 85.5 8765 84.1Previous gonorrhoea diagnosisYes 619 5.9 310 3.0No 9819 94.1 10 115 97.0Total 10 437 100.0 10 425 100.0*Imputed values. Q11Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992 3Clinical257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384Please do not annotate this PDF with corrections - use the unmarked copy providedgeographical validation population was 0.69, 95% CI 0.67 to 0.71,while the AUC in the derivation population was 0.74, 95% CI 0.70to 0.77 (online supplementary figure 3). A p value of 0.26 for theHosmer–Lemeshow goodness-of-fit test also indicated good cali-bration. Online supplementary figure 4 shows the calibration in thegeographic validation population was good as the prevalence ofchlamydia and/or gonorrhoea infection increased with increasingrisk score, which ranged from 0.2% in the lowest risk score cat-egory to 23.7% in the highest risk category. We also explored theuse of the risk score for selective screening (table 4). This analysisidentified a risk score cut-off level of ≥6 points that would identifyapproximately 95% of infections while screening 78% of the geo-graphical validation population. In the derivation population, thesame risk score cut-off of ≥6 points identified 91% of cases andthe fraction screened was 68% of the population.CONCLUSIONSValidation studies aim to provide evidence that a risk scoringalgorithm can be generalised to new populations. The‘Vancouver’ risk estimation tool showed slightly better discrim-ination in the geographical validation population (AUC=0.69)than in the temporal validation population (AUC=0.64). Therisk estimation tool performed well in the geographical valid-ation population despite the fact that the geographical validationpopulation differed from the derivation population regardingsome predictors (eg, age, condom use and previous infection).The geographic validation regions have higher rates of chla-mydia and gonorrhoea infection and dissimilar STI epidemi-ology and social determinants of sexual health;21–23 thetemporal validation population also was less heterogeneous thanthe derivation and geographical validation populations. FurtherTable 3 Chlamydia and/or gonorrhoea prevalence and unadjusted ORs (derivation and geographic validation populations)*Derivation population (N=10 437) Geographical validation population (N=10 425)Variable % OR (95% CI) % OR (95% CI)GenderFemale 2.1 1.31 (0.97 to 1.77) 6.8 1.67 (1.40 to 1.98)Male 1.6 Ref 4.2 RefAge (years)14–19 7.4 6.49 (3.58 to 11.75) 9.7 9.47 (6.00 to 14.94)20–24 2.8 2.30 (1.46 to 3.63) 7.7 7.40 (4.72 to 11.58)25–29 1.8 1.50 (0.94 to 2.38) 4.9 4.55 (2.82 to 7.34)30–39 1.1 0.88 (0.53 to 1.45) 2.4 2.15 (1.27 to 3.62)≥40 1.2 Ref 1.1 RefRace/ethnicityWhite 1.2 Ref 5.2 RefNon-white 3.4 2.90 (2.16 to 3.89) 5.7 1.10 (0.78 to 1.55)No. of sexual partners in previous six months0 0.5 Ref 2.0 Ref1–2 1.8 3.43 (1.10 to 10.73) 4.9 2.50 (1.26 to 4.98)≥3 2.0 3.92 (1.23 to 12.45) 7.1 3.71 (1.83 to 7.55)Condom useNever 1.6 Ref 4.5 RefSometimes 2.0 1.22 (0.84 to 1.77) 6.0 1.36 (1.07 to 1.72)Always 1.4 0.88 (0.56 to 1.39) 5.0 1.11 (0.85 to 1.46)Sex with partners recruited onlineYes 1.2 0.67 (0.27 to 1.63) 2.7 0.48 (0.25 to 0.94)No 1.8 Ref 5.4 RefInjection drug useYes 1.6 0.89 (0.26 to 3.06) 7.5 1.45 (0.81 to 2.58)No 1.8 Ref 5.3 RefSex with injection drug userYes 1.0 0.55 (0.20 to 1.54) 6.6 1.27 (0.78 to 2.07)No 1.8 Ref 5.3 RefSex with commercial sex workerYes 1.2 0.65 (0.35 to 1.20) 2.7 0.48 (0.29 to 0.79)No 1.8 Ref 5.6 RefPrevious chlamydia diagnosisYes 5.1 4.40 (3.27 to 5.93) 11.7 3.10 (2.58 to 3.72)No 1.2 Ref 4.1 RefPrevious gonorrhoea diagnosisYes 2.1 1.21 (0.69 to 2.14) 4.2 0.77 (0.44 to 1.36)No 1.7 Ref 5.4 RefTotal 1.8 – 5.3 –*Missing data among the predictor variables were handled using a multiple imputation procedure with five resampling replications, which generated an augmented database with(5×10 437) 52 185 and (5×10 425) 52 125 observations with complete data in the derivation and temporal validation populations, respectively. With the imputed sample, we estimatedbaseline characteristics and developed prediction models. The average of all five imputed samples is shown in this table.4 Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992Clinical385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512Please do not annotate this PDF with corrections - use the unmarked copy providedanalysis also revealed that a remodelled algorithm using datafrom the geographical validation population performed nobetter than the ‘Vancouver’ prediction algorithm as indicated bythe non-significant χ2 statistic testing the difference between theAUCs. These findings provide strong evidence that the riskscore is robust and valid and likely has generalisable discrimin-ation and calibration in varied settings.In the geographical validation population, choosing the‘Vancouver’ cut-off point of ≥6 would require screening 78% ofthe population to find 95% of the cases and equates to a reduc-tion of 22% in the number of individuals that would need to bescreened. However, using the ≥6 cut-off point would fail tomeet the efficiency benchmark of screening <60% of the popu-lation. Increasing the cut-off point to ≥7 would require screen-ing 67% of the population to achieve a sensitivity of 90%,which would be closer to the efficiency benchmark. In applyingthe prediction rule to a population with a higher prevalence ofinfection and risk behaviours (ie, more severe case mix) such asthe geographical validation population, it is expected thatchoosing a higher cut-off point will increase the efficiency ofscreening decision-making.24 25Alternatively in this setting, applying the age-based screeningcriterion (ie, age <25 years) to the geographical validationpopulation would require screening of 45% of the populationbut would only detect 71% of the cases, a performance thatfalls short of the screening benchmark. However, increasing thecut-off to age <30 years would require screening 64% of thepopulation with a sensitivity of 88%, a performance that isclose to the benchmark and also similar to using the cut-offpoint of ≥7. This finding suggests that using age alone could bea viable option as a screening criterion in the geographical valid-ation population, but not in the derivation population. Thisfinding was not surprising because the distribution of age in thegeographical validation population was more heterogeneousthan in the derivation population—a situation that often leadsto better discrimination and optimum screening performance.25However, age-based screening criterion may be contraindicatedin settings where a majority of the population presenting forscreening is comprised of younger individuals (eg, youthclinics). Also, if the prediction rule is found to perform less thanadequately in these settings, universal screening may be a moresuitable alternative.Several studies have established the validity of prediction rulesas screening criteria, especially where chlamydia and/or gonor-rhoea prevalence is low (ie, <2%), as is the case in the deriv-ation population used in this study.26 Although it has beensuggested that universal screening or using criteria based on agewould be cost effective in settings with prevalence of 2% ormore, publicly funded sexual health services in these settings areconstrained by available funding and limited resources. In con-sidering such practical constraints within the BC, we suggest acautious approach to such global screening approaches.26Table 4 Sensitivity and specificity of cut-off scores in the derivation and geographical validation populationsDerivation population (%) Geographical validation population (%)Score cut-off Sensitivity Specificity Fraction screened PPV Sensitivity Specificity Fraction screened PPV≥−2 100.0 0.0 100.0 1.8 100.0 0.0 100.0 5.3≥−1 100.0 1.2 98.8 1.8 100.0 0.7 99.3 5.4≥0 100.0 1.3 98.7 1.8 100.0 0.7 99.3 5.4≥1 100.0 2.6 97.4 1.8 99.9 1.7 98.4 5.4≥2 99.5 3.7 96.4 1.8 99.9 2.5 97.7 5.5≥3 99.5 3.7 96.4 1.8 99.9 2.5 97.7 5.5≥4 96.7 16.7 83.5 2.0 97.1 10.8 89.7 5.8≥5 95.8 22.2 78.1 2.2 96.2 13.7 86.9 5.9≥6 91.2 32.7 67.8 2.4 94.5 22.5 78.4 6.4≥7 84.9 47.7 52.8 2.8 90.1 34.1 67.2 7.2≥8 82.2 53.0 47.7 3.0 86.0 37.9 63.3 7.2≥9 72.2 65.2 35.4 3.6 71.6 52.9 48.4 7.9≥10 67.3 70.6 30.1 3.9 64.0 58.9 42.3 8.1≥11 62.5 74.8 25.8 4.3 61.8 63.0 38.4 8.6≥12 52.6 81.4 19.2 4.8 59.3 67.4 34.1 9.3≥13 47.7 84.5 16.1 5.2 57.6 69.5 31.9 9.6≥14 35.1 91.1 9.3 6.6 41.5 83.1 18.2 12.1≥15 32.4 94.2 6.3 9.1 32.4 88.2 12.9 13.4≥16 25.9 95.9 4.5 10.1 25.6 91.1 9.8 13.9≥17 21.2 96.9 3.4 10.9 22.4 92.4 8.4 14.3≥18 19.9 97.2 3.1 11.4 21.8 92.8 8.0 14.5≥19 14.3 98.5 1.8 14.4 17.4 95.9 4.9 19.1≥20 11.6 99.0 1.2 16.9 14.8 96.9 3.7 21.2≥21 7.3 99.5 0.6 20.9 8.1 98.5 1.8 23.7≥22 2.8 99.9 0.2 28.0 3.5 99.4 0.7 26.1≥23 2.8 99.9 0.1 33.8 3.4 99.5 0.7 26.3≥24 2.8 99.9 0.1 33.8 3.4 99.5 0.7 26.3≥25 2.8 99.9 0.1 33.8 3.4 99.5 0.7 26.3≥26 0.7 100.0 0.0 30.0 1.5 99.8 0.3 28.1≥27 0.1 100.0 0.0 18.2Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992 5Clinical513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640Please do not annotate this PDF with corrections - use the unmarked copy providedSpecifically, screening women <25 years old could prove to becost prohibitive in settings where individuals in this age groupcomprise the highest proportion of clinic visits.27 Cautionshould also be exercised before using age as a criterion ininternet-based testing scenarios such as GCO where good cali-bration (and not just discrimination) would be essential. In thesescenarios, the risk score categories and their associated preva-lence prove more useful than age-based criterion for patientstrying to decide whether to take the STI test. This is because theprocess of moving from screening criteria that focus on specificrisk factors (eg, age) to prediction rules acknowledges a morecomprehensive risk spectrum.28The findings of this analysis also were compared with otherexternal validation studies that examined the validity of previ-ously derived clinical prediction rules (CPRs) in new geographicsettings.24 29 Gotz and colleagues derived a prediction rule forchlamydia infection for the selective screening of high-risk indi-viduals in Rotterdam, the Netherlands.24 The prediction ruleshowed fair external validity in two independent settings: apopulation-based study in Amsterdam and an outreach screeningproject among high-risk youth in Rotterdam. The AUC was0.79 (95% CI 0.76 to 0.84) in the derivation sample, 0.66(95% CI 0.58 to 0.74) in the Amsterdam sample and 0.68(95% CI 0.58 to 0.79) in the Rotterdam sample.24 A secondstudy by Haukoos et al29 derived and validated an algorithm toaccurately identifyQ8 patients at risk for HIV infection, usingpatient data from an STI clinic in Denver, Colorado (1996–2008). Validation was performed using an independent popula-tion from an urban emergency department in Cincinnati, Ohio.The results of the study showed that the risk score showed rea-sonable generalisability; the AUC was 0.85 (95% CI 0.83 to0.88) in the derivation sample and 0.75 (95% CI 0.70 to 0.78)in the validation sample.29The AUC of the ‘Vancouver’ risk estimation in the derivationpopulation was lower compared with the AUCs of the twoaforementioned studies.24 29 This can be explained by the omis-sion of symptoms in the ‘Vancouver’ risk estimation tool, whichhave been shown to be significantly associated with infection.30Specifically, unlike previous risk estimation tools in sexual healthsettings, the ‘Vancouver’ risk estimation tool was limited toasymptomatic patients, an important improvement as most STIsinfrequently present with symptoms. The loss in discriminativeability between the derivation and validation populations in theother studies ranged (in absolute percentage points) from 10%points to 13% points compared with a loss of 10% points and5% points in the temporal and geographical validation popula-tions, respectively, in this analysis.There were several strengths to the geographic validationprocess undertaken here, including the large overall populationsize, the independence of the clinicians in the geographical val-idation population from the derivation population, and the sys-tematic analysis of its discrimination and calibration. Thecurrent study was the first to derive and validate a locally spe-cific risk assessment tool to quantify STI risk in a Canadiansetting. Risk assessment tools ideally should be derived fromlarge representative samples.31 This study included 13 years ofelectronic health records comprising 40 000 patient visits topublicly funded STI clinics in BC, representing a high percent-age of the population of individuals using this service in theprovince. As with most administrative datasets, the dataset wasnot deliberately built for the derivation of risk algorithms,resulting in some missing information for several predictors,which we have attempted to mitigate through the use of imput-ation (something rarely done in prediction modelling studies).The use of imputation techniques yielded discrimination andcalibration performance measures similar to those of completecase analyses in which individuals with missing values on any ofthe considered variables were excluded and baseline analyses inwhich individuals with missing values on a variable wereassumed to be in the lowest category (data not shown).15 Thisfinding suggests that the algorithm was valid despite the conse-quential risk factor misclassification associated with the dataimputation process. Overall, however, the use of imputationtechniques offers improved study power and limited bias in theestimated regression coefficients.15Caution should be exercised in generalising the findings ofthis analysis to even more diverse geographic settings. Severaladditional analyses are recommended before the widespreadimplementation of the risk estimation algorithm. For example,the algorithm’s screening performance could be prospectivelyverified in internet-based STI testing contexts. Furthermore,while the derivation and validation populations are an unbiasedrepresentation of STI clinic clients in BC, the current resultsmight or might not be valid for other settings in BC (or otherCanadian provinces, or even other global settings). It would bereasonable to argue that STI clinic clients also may vary signifi-cantly from patients seeking care in primary care settings oronline contexts; and, therefore, the results of the current CPRshould not be directly extrapolated to other settings withoutadditional validation studies that could provide stronger evi-dence for the generalisability of the risk estimation algorithm.In conclusion, a new era in evidence-based decision-makingregarding STI testing and progress in relation to the adoption ofprediction rules may be at hand. The advent of onlineapproaches to risk estimation, the emergence of new statisticalmethods, as well as increasingly sophisticated theory, all reflectthe potential to continue to make advanced in improving testingand treatment of STIs. To date, however, few prediction ruleshave been validated and, hence, the dissemination and usage ofprediction rules in STI service provision remains in the nascentstages. The well-performing prediction rule derived and broadlyvalidated here provides evidence that risk estimation tools havea place in sexual health service provision. New investments inresearch and practice are required to facilitate the effective inte-gration of prediction rules into routine sexual health serviceprovision and more attention should be paid to their scaling upand to the scientific evaluation of their effects over time.Key messages▸ This article highlights the geographical validation of the‘Vancouver’ risk estimation tool for screening asymptomaticchlamydia and gonorrhoea.▸ The prediction tool showed adequate discrimination andcalibration upon validation in seven clinics outside ofVancouver.▸ These findings are encouraging and bolster confidence inrecommending this tool for use in sexual health services andprogrammes.▸ The risk score could be easily implemented and is accurateenough to convey important screening considerations.Handling editor Jackie A CassellTwitter Follow Mark Gilbert at @mpjgilbert6 Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992Clinical641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768Please do not annotate this PDF with corrections - use the unmarked copy providedContributors TF was the lead investigator for the empirical work presented andresponsible for all major areas of concept formation, data analysis, as well asmanuscript composition. MG and PG were involved in the early stages of conceptformation and contributed to manuscript edits. PG also contributed data analysesand interpretation. JS was the supervisory author on this project and was involvedthroughout the project in concept formation and manuscript composition.Q5Funding TF¶was supported by the Canadian Institutes of Health Research (CIHR)Doctoral Research Award. PG¶was supported by a grant from the Natural Sciencesand Engineering Research Council of Canada.Competing interests None declared.Ethics approval University of British Columbia’s Research Ethics Board (certificate# H11-02000).Provenance and peer review Not commissioned; externally peer reviewed.REFERENCES1 Chai SJ, Aumakhan B, Barnes M, et al. Internet-based screening for sexuallytransmitted infections to reach nonclinic populations in the community: Risk factorsfor infection in men. Sex Transm Dis 2010;37:756–63.2 Gaydos CA, Dwyer K, Barnes M, et al. Internet-based screening for chlamydiatrachomatis to reach non-clinic populations with mailed self-administered vaginalswabs. Sex Transm Dis 2006;33:451–7.3 Shamos SJ, Mettenbrink CJ, Subiadur JA, et al. Evaluation of a testing-only“express” visit option to enhance efficiency in a busy STI clinic. Sex Transm Dis2008;35:336–40.4 Hottes TS, Farrell J, Bondyra M, et al. Internet-based HIV and sexually transmittedinfection testing in british columbia, canada: opinions and expectations ofprospective clients. J Med Internet Res 2012;14:e41.5 Falasinnu T, Gilbert M, Gustafson P, et al. Deriving and validating a risk estimation toolfor screening asymptomatic chlamydia and gonorrhea. Sex Transm Dis 2014;41:706.6 Gilbert M. Impact of an integrated internet-based testing program for HIV andsexually transmitted infections in british columbia. CIHR’s Partnership for HealthSystem Improvement Grant. 2010(350,000 CAD).7 Knottnerus J, Buntinx F, eds. The evidence base of clinical diagnosis: theory andmethods of diagnostic research. 2009Q9 .8 D’Agostino RB Sr, Grundy S, Sullivan LM, et al. CHD Risk Prediction Group.Validation of the framingham coronary heart disease prediction scores: Results of amultiple ethnic groups investigation. JAMA 2001;286:180–7.9 Siontis GC, Tzoulaki I, Siontis KC, et al. Comparisons of established risk predictionmodels for cardiovascular disease: Systematic review. BMJ 2012;344:e3318.10 Moons KG, Kengne AP, Grobbee DE, et al. Risk prediction models: II. externalvalidation, model updating, and impact assessment. Heart 2012;98:691–8.11 Toll DB, Janssen KJ, Vergouwe Y, et al. Validation, updating and impact of clinicalprediction rules: a review. J Clin Epidemiol 2008;61:1085–94.12 Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognosticinformation. Ann Intern Med 1999;130:515–24.13 Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled? issues in theuse of risk indexes for assessing mortality after coronary artery bypass graft surgery.Circulation 1999;99:2098–104.14 Falasinnu T, Gustafson P, Gilbert M, et al. Risk prediction in sexual health contexts:Protocol. JMIR Res Protoc 2013;2:e57.15 Vergouwe Y, Royston P, Moons KG, et al. Development and validation of aprediction model with missing predictor data: A practical approach. J Clin Epidemiol2010;63:205–14.16 Janssen KJ, Vergouwe Y, Donders AR, et al. Dealing with missingpredictor values when applying clinical prediction models. Clin Chem2009;55:994–1001.17 Survey Research Center, Institute for Social Research, University of Michigan.IVEware: Imputation and Variance Estimation software. Updated 2013. http://www.isr.umich.edu/src/smp/ive/ (accessed 8 jul, 2013).18 Sanagou M, Wolfe R, Leder K, et al. External validation and updating of aprediction model for nosocomial pneumonia after coronary artery bypass graftsurgery. Epidemiol Infect 2014;142:540–4.19 La Montagne DS, Patrick LE, Fine DN, et al. Infertility Prevention Project.Re-evaluating selective screening criteria for chlamydial infection among women inthe U S pacific northwest. Sex Transm Dis 2004;31:283–9.20 DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two ormore correlated receiver operating characteristic curves: A nonparametric approach.Biometrics 1988;44:837–45.21 Goldenberg S, Shoveller J, Koehoorn M, et al. Barriers to STI testing among youthin a canadian oil and gas community. Health Place 2008;14:718–29.22 Goldenberg S, Shoveller J, Ostry A, et al. Youth sexual behaviour in a boomtown:Implications for the control of sexually transmitted infections. Sex Transm Infect2008;84:220–3.23 Shoveller J, Johnson J, Rosenberg M, et al. Youth’s experiences with STI testingin four communities in british columbia, canada. Sex Transm Infect2009;85:397–401.24 Gotz HM, Veldhuijzen IK, Habbema JD, et al. Prediction of chlamydia trachomatisinfection: Application of a scoring rule to other populations. Sex Transm Dis2006;33:374–80.25 Ewout W. Steyerberg. Clinical prediction models: a practical approach todevelopment, validation, and updating. New York: Springer, 2009.26 Han Y, Coles FB, Hipp S. Screening criteria for chlamydia trachomatis in familyplanning clinics: accounting for prevalence and clients’ characteristics. Fam PlannPerspect 1997;29:163–6.27 Miller WC, Hoffman IF, Owen-O’Dowd J, et al. Selective screening for chlamydialinfection: which criteria to use? Am J Prev Med 2000;18:115–22.28 Eddy DM, Adler J, Patterson B, et al. Individualized guidelines: the potential forincreasing quality and reducing costs. Ann Intern Med 2011;154:627–34.29 Haukoos JS, Lyons MS, Lindsell CJ, et al. Derivation and validation of the denverhuman immunodeficiency virus (HIV) risk score for targeted HIV screening. Am JEpidemiol 2012;175:838–46.30 Xu F, Stoner BP, Taylor SN, et al. “Testing-only” visits: an assessment of misseddiagnoses in clients attending sexually transmitted disease clinics. Sex Transm Dis2013;40:64–9.31 Wand H, Guy R, Donovan B, et al. Developing and validating a risk scoring tool forchlamydia infection among sexual health clinic attendees in australia: A simplealgorithm to identify those at high risk of chlamydia infection. BMJ Open 2011;1:e000005.Falasinnu T, et al. Sex Transm Infect 2015;0:1–7. doi:10.1136/sextrans-2014-051992 7Clinical769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896Please do not annotate this PDF with corrections - use the unmarked copy provided


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items