Development and validation of a predictive model for endocervical curettage in patients referred for colposcopy: A multicenter retrospective diagnostic study in China

Peng Xue ,Bingrui Wei ,Samuel Seery ,Qing Li ,Zichen Ye ,Yu Jiang ,Youlin Qiao

1Department of Epidemiology and Biostatistics,School of Population Medicine and Public Health,Chinese Academy of Medical Sciences and Peking Union Medical College,Beijing 100730,China;2 Faculty of Health and Medicine,Division of Health Research,Lancaster University,Lancaster,LA1 4YW,United Kingdom;3 Diagnosis and Treatment for Cervical Lesions Center,Shenzhen Maternity and Child Healthcare Hospital,Shenzhen 518028,China

Abstract Objective:This study aimed to develop a nomogram that can predict occult high-grade squamous intraepithelial lesions or worse (HSIL+) and determine the need for endocervical curettage (ECC) in patients referred for colposcopy.Methods:This retrospective multicenter study included 4,149 patients who were referred to any one of six tertiary hospitals in China for colposcopy between January 2020 and November 2021 because of abnormal screening results.ECC data were extracted from the medical records.Univariate and multivariate logistic regression analyses were performed to identify factors that could predict HSIL+on ECC.Patients were randomly assigned to a training set or to an internal validation set for performance and comparability testing.The model was externally validated and tested in patients from two additional hospitals.The nomogram was assessed in terms of discrimination and calibration and subjected to decision curve analysis.Results:HSIL+was found on ECC in 38.8% (n=388) of cases.Our predictive nomogram included age group,cytology,human papillomavirus (HPV) status,visibility of the cervix and colposcopic impression.The nomogram had good overall discrimination,which was internally validated [area under the receiver-operator characteristic(AUC),0.839;95% confidence interval (95% CI),0.773-0.904].In terms of external validation,the AUC was 0.843(95% CI,0.773-0.912) for the consecutive sample and 0.843 (95% CI,0.783-0.902) for the comparative sample.Calibration analysis suggested good consistency between predicted and observed probabilities.Decision curve analysis suggested this nomogram would be clinically useful with almost the entire range of threshold probabilities.Conclusions:This internally and externally validated nomogram can be easily applied and incorporates multiple clinically relevant variables that can be used to identify patients with occult HSIL+who need ECC.

Keywords: Cervical cancer;colposcopy;endocervical curettage;nomogram

Introduction

Millions of females have abnormal screening results and undergo colposcopy each year worldwide (1-4).Clinicians are often required to perform endocervical curettage(ECC) to help rule out an occult,high-grade squamous intraepithelial or worse lesion (HSIL+).This additional procedure provides information that cannot be obtained by standard visual colposcopic examination.However,the inter-observer agreement between ECC and histopathologic interpretation is relatively low (5,6).Furthermore,ECC can cause discomfort,cramping and even sharp pain.Moreover,the international clinical practice guidelines are presently unclear in terms of the indication for ECC at the time of colposcopy.Therefore,not all females benefit from ECC (7,8) and there is a need to develop a strategy that can predict occult HSIL+in patients referred for colposcopy.

At present,there is widespread acceptance that ECC has benefits in older females,mainly because the cervix is often partially or completely invisible in this age group (9).The results of studies that have sought to identify specific subgroups of females who are referred for colposcopy and need ECC have been inconclusive,and findings based on human papillomavirus (HPV) testing alone,cytology and colposcopic impression have not been consistent.Possible reasons for these inconsistent results include sample sizes that were too small for adequate statistical power and lack of ethnic homogeneity.Either way,clinicians are left to decide whether to perform ECC based mainly on past experience.Moreover,ECC is a traumatic procedure that can discourage females from attending for follow-up or participating in other health screening programs (10).Therefore,decisions about whether or not to perform ECC should not be taken lightly,and there is a need for an alternative approach to allow early identification of females in whom this procedure is required.

A nomogram is a graphic calculator that allows individualized predictions to be made in clinical practice(11,12).A nomogram can intercalate any number of variables,and best practice stipulates that the process of development begins by identifying statistically significant risk factors.HSIL+is prevalent in most if not all societies but is often occult.Therefore,it is necessary to expand our knowledge of risk factors.It may even be possible to develop and validate a nomogram that renders ECC obsolete.The aim of this study was to construct a reliable nomogram that can provide an individualized estimate of the risk that occult HSIL+will be detected by ECC.We hope that our findings in a relatively homogeneous sample of Chinese females will prompt comparative studies in other ethnic populations and lead to development of a nomogram that can be applied internationally.

Materials and methods

Study design and participants

This retrospective,multicenter,diagnostic study analyzed anonymized data collected from the electronic medical records of six hospitals across mainland China between January 2020 and November 2021.These data included demographics,medical history,cytology,HPV status,colposcopic impression and results of ECC.Data from four municipal or provincial hospitals were used for training and internal validation of our predictive model.There was no overlap between the training set and the internal validation set.Digital records from consecutive patients were also obtained from two additional hospitals to determine the diagnostic performance of our nomogram.These data were grouped and defined as external validation sets 1 and 2.

All study participants had been referred because of abnormal screening results and underwent routine colposcopy with ECC.Patients were excluded if they had a history of ablation,cryosurgery,or pelvic radiotherapy involving the cervix,if endocervical sampling was unsatisfactory,or if data were incomplete.Abnormal screening results included cytology and HPV testing.

Cytology results were reported based on the revised Bethesda nomenclature (13) as negative for intraepithelial lesions or malignancy (NILM),atypical squamous cells of undetermined significance (ASC-US),low-grade squamous intraepithelial lesion (LSIL),atypical squamous cells that cannot exclude high-grade squamous intraepithelial lesion(ASC-H) and high-grade squamous intraepithelial lesion(HSIL).

HPV status was defined as high-risk (HPV 16/18 or other high-risk non-16/18 HPV) or negative.Type of HPV was not recorded because it was not deemed to be relevant to this study.All colposcopic examinations were performed by experienced colposcopists and included visual assessment of the cervix,including its visibility,type of transformation zone (TZ) and colposcopic impression.All cases in which the colposcopic impression was abnormal were biopsied.

The ECC procedures were performed using a Kevorkian curette.Pathological diagnoses based on ECC were reviewed by experienced pathologists at local hospitals.Any disagreements were resolved by discussion.ECC results were classified as normal,LSIL,HSIL,or invasive cancer according to the Lower Anogenital Squamous Terminology (LAST) system.The worst grade of dysplasia present was considered as the final diagnosis.HSIL+cases detected by ECC,including HSIL and invasive cancer,were used as the “ground truth” for training and validation purposes.

The study was approved by the Institutional Review Board of the Chinese Academy of Medical Sciences and Peking Union Medical College (No.CAMS &PUMCIEC-2022-022) and performed in accordance with the tenets of the Declaration of Helsinki.The requirement for written informed consent was waived in view of the retrospective observational nature of the research and anonymity of the data.

Selection of predictors

All predictors were selected on the basis of a detailed literature review and clinical evidence and were constrained only by data availability.All risk factors reported to contribute to detection of HSIL+by ECC were used.Given that the model is intended for use in clinical practice,the following 10 predictors were considered when developing the prediction model.The coding procedures for these variables are shown inSupplementary Table S1.

Development and validation of the model

The model was developed and validated according to the Transparent Reporting of a Multi-variable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD)guidelines (14).The complete development dataset was randomly sampled according to the distribution of ECC results (i.e.,HSIL+vs.＜HSIL).Individuals were then assigned to the training set or to the internal validation set to assess performance at a ratio of 8:2.

Using the training set,we selected variables that were identified as statistically significant according to simple logistic regression analysis of ECC HSIL+probabilities.For each combination of candidate predictors,we generated a candidate model by backward elimination using the Akaike information criterion and Bayesian information criterion.We selected the model with the lowest Akaike and Bayesian values (1,153.97 and 1,189.06,respectively) as the final logistic regression model,which included five predictors (i.e.,age group,results of cytology,HPV status,visibility of cervix and colposcopic impression).All included variables were subjected to univariate analysis.Factors with P＜0.05 in univariate analysis were included in multivariate logistic analysis.Correlations are described using the odds ratio (OR) and corresponding 95% confidence interval(95% CI).

To assess the performance of the model during development and throughout internal and external validation,we calculated the ability of the model to produce unbiased estimates.We also calculated its discrimination ability and generated model decision curves.Model calibration was assessed visually using calibration plots,which intercalate predicted plots versus the observed results.Model discrimination was determined by analysis of area under the receiver-operating characteristic (AUC)curves.Decision curve analysis was performed,whereby a higher net benefit indicated enhanced clinical usefulness of the model.

Presentation of the model

The validated model,which included statistically significant variables,was presented as a regression equation and converted into a nomogram.Each predictor in the nomogram was assigned a regression weight so that the total score was equivalent to a linear predictor.For this model,logistic transformation was applied to the linear predictor to produce estimates of the probability of ECC HSIL+.

Statistical analysis

The sample size was based on the availability of data because there is presently no standard method available for pre-calculation of sample size.There were 10 candidate predictors and over 35 events per variable.Histological diagnoses were provided using ECC samples.Receiveroperating characteristic (ROC) curves were created by plotting the true positive rates (i.e.,sensitivity scores)against the false positive rates (i.e.,1-specificity).AUC values were generated for further comparative analysis.Univariate and multivariate logistic regression analyses were performed using Stata software (Version 15.0;StataCorp LLC,College Station,TX,USA).R software(Version 3.6.1;R Foundation for Statistical Computing,Vienna,Austria) was used for nomogram calculations and for validation.All statistical tests were two-sided,and P＜0.05 was considered statistically significant.

Results

Patient demographic and clinical characteristics

We identified 3,811 patients who had undergone colposcopy with ECC during the study period,303 of whom were excluded according to our predefined selection criteria.Data for 3,508 patients were used to develop the nomogram and for internal validation.The training set consisted of 250 HSIL+cases and 2,556 ＜HSIL controls.A flow diagram summarizing the patient selection and assignment process is shown inFigure 1.Both external validation datasets were based on information collected between April and November,2021;341 consecutive patients and 300 patients who had undergone ECC were enrolled.Table 1shows the demographic and clinical characteristics of the study sample.

Figure 1 Flow chart showing the process used to develop and validate the predictive model.ECC,endocervical curettage;HSIL+,highgrade squamous intraepithelial lesion or worse.

Univariate and multivariate predictors of ECC HSIL+

Univariate analysis identified 10 candidate predictors that were significantly associated with ECC HSIL+,namely,age group,gravidity,parity,menopausal status,cytology,HPV status,visibility of the cervix,type of TZ,colposcopic impression and size of the lesion.In multivariate logistic regression analysis,age group,cytology,HPV status,visibility of the cervix and colposcopic impression remained significant predictors of ECC HSIL+.The risk of ECC HSIL+in females older than 50 years was found to be 2.628 (95% CI,1.008-6.854) times higher than that for females younger than 30 years (P=0.048).

The risk of ECC HSIL+was 1.789 (95% CI,0.995-3.214) times higher in patients with ASC-H than in those patients with negative cytology,but there was no statistical significance (P=0.052).Patients with cytology HSIL had 2.648 (95% CI,1.649-4.252) times significantly higher risk of ECC HSIL+than those in whom it was negative (P＜0.001).Patients who were HPV 16/18-positive had a 3.403 (95% CI,1.501-7.714) (P=0.003) times higher risk of ECC HSIL+than those who were HPV-negative.Patients in whom the colposcopic impression was high grade had a 22.747 (95% CI,9.854-52.512) (P＜0.001)times higher risk of ECC HSIL+than those in whom it was normal/benign.Detailed information is provided inSupplementary Table S2.

Table 1 Demographics and clinical characteristics of study population

Predictor selection and model development

The final model included five predictors,namely,age group,cytology,HPV status,visibility of the cervix and colposcopic impression (Table 2).Supplementary Figure S1provides the weighted nomogram and risk prediction scores that allowed estimation of individual probabilities of identifying ECC HSIL+at colposcopy.The probability of identifying ECC HSIL+was then identified using a total point scale.

Table 2 Included predictors in final ECC prediction model

Discrimination,calibration and decision curve analysis

The model showed good discrimination accuracy with an AUC of 0.885 (95% CI,0.863-0.906) in the training set and 0.839 (95% CI,0.773-0.904) in the internal validation set.In both external validation sets,the AUCs for the model were 0.843 (95% CI,0773-0.912) and 0.843 (95%CI,0.783-0.902),respectively.

Overall,AUC analysis suggested good generalizability of the prediction model in all external datasets.ROC curves for the model in the different datasets are shown inFigure 2.Supplementary Table S3shows the performance indices [i.e.,sensitivity,specificity,accuracy,positive predictive value (PPV),and negative predictive value(NPV)] for each probability as a cut-off value for identifying individuals at risk of ECC HSIL+.The resulting model prediction scores from 0-1 were categorized to balance sensitivity and specificity for detection.

Table S1 Coding variables

Table S2 Univariate and multivariate logistical regression analysis of predictors associated with ECC HSIL+

Table S3 Performances of each probability as a cut-off value for identifying individuals at a high risk of ECC HSIL+

Figure 3shows calibration plots for the observed frequencies and predicts probabilities through training and across the internal and external validation sets.Good calibration was observed based on the comparable intercepts for predictive models in the training and internal validation sets.This suggested that the model had high stability and a low level of over-fitting.

Figure 4summarizes the decision curve analyses across the training set and the internal and external validation sets.The solid blue line represents the net clinical benefit of performing ECC in all cases and the horizontal black dotted line represents the net benefit of not performing ECC.The horizontal and vertical axes on this figure represent the threshold probability,and the net benefit after advantages were subtracted according to disadvantages.

Figure S1 Nomogram for ECC HSIL+risk.ECC,endocervical curettage;HSIL+,high-grade squamous intraepithelial lesion or worse;HPV,human papillomavirus;NILM,negative for intraepithelial lesion or malignancy;ASC-H,atypical squamous cells which did not exclude high-grade squamous intraepithelial lesion;ASC-US,atypical squamous cells of undetermined significance;LSIL,low-grade squamous intraepithelial lesion;HSIL,high-grade squamous intraepithelial lesion;hr-HPV,high-risk human papillomavirus.

Figure 2 Discrimination performance of predictive model in training,internal and external validation sets.Traning set: AUC 0.885 (95% CI,0.863-0.906);Internal validation set: AUC 0.839(95% CI,0.773-0.904);External validation set 1: AUC 0.843(95% CI,0.773-0.912);External validation set 2: AUC 0.843(95% CI,0.783-0.902).AUC,area under the receiver-operator characteristic;95% CI,95% confidence interval.

Figure 3 Calibration plots showing the observed frequency and predicted probability for the predictive model in training set (N=2,806,E:O=1.000,CITL=-0.000,slope=1.000,AUC=0.885) (A),internal validation set (N=702,E:O=0.979,CITL=0.033,slope=0.826,AUC=0.839) (B),external validation set 1 (N=341,E:O=0.706,CITL=0.640,slope=0.667,AUC=0.843) (C) and external validation set 2(N=300,E:O=0.873,CITL=0.217,slope=0.679,AUC=0.843) (D).E:O,the observed divided by expected number,with a number close to 1 showing good model fit;CITL,calibration-in-the-large;AUC,area under the receiver-operator characteristic.

Figure 4 Decision curve analysis showing the net benefit derived from training set (A),internal validation set (B),external validation set 1(C) and external validation set 2 (D).The horizontal and ordinate axis of this figure represents the threshold probability,and the net benefit after the advantages was subtracted by the disadvantage,respectively.When a patient’s risk of HSIL+reached a certain threshold,it was defined as high-risk and ECC was performed to confirm the final diagnosis.Decision curve analysis showed higher net benefit than ECC for all patients,which suggests that the model developed in this study is clinical useful.ECC,endocervical curettage;HSIL+,high-grade squamous intraepithelial lesions or worse.

In the training and internal validation sets,our model increased the overall net clinical benefit when the threshold probability was ＜70% and improved diagnostic performance compared with omitting ECC across all sets.At a threshold probability of around 10%,using our model would result in 6.4 additional significant HSIL+detected per 100 patients compared with not performing ECC.Similar net benefits were also observed in the external validation sets,which indicates that the model is potentially clinically useful.

Discussion

Although ECC is becoming part of routine colposcopic practice,its value remains controversial because of lack of consistent findings (15).Moreover,the procedure is invasive and has a ripple effect in terms of the willingness of females to attend for follow-up.Clinicians in China often prefer to perform ECC to avoid missing occult HSIL+cases,even in unlikely instances,despite knowing that not all females will benefit (16) and the financial implications of excessive testing.Previous studies have found that when ECC is performed in all suspected cases,the diagnostic rate for HSIL+increases by only about 1%(17,18).However,we do not have a cost-effective method for predicting occult HSIL+during colposcopy.Therefore,in this study,we identified statistically significant predictors of ECC HSIL+and developed a nomogram that individualizes the risk of ECC HSIL+in patients referred for colposcopy.

We identified five significant predictors,namely,age group,cytology,HPV status,visibility of the cervix,and colposcopic impression,which contributed to development of our prediction model.The most common reasons for referral to a colposcopist are abnormal cervical cytology and HPV infection.Therefore,it is reasonable that severe HPV infection,cytology and colposcopic impression were found to be significant variables for determining whether ECC should be performed.The findings of our multivariate logistic regression analysis are in line with previous clinical reports of strong correlations of ECC HSIL+with HPV16/18 positivity,ASC-H or HSIL cytology,and a high-grade colposcopic impression(9,19,20).However,the probability of occult HSIL+in the endocervix varies according to age and visibility of the cervix (9,21).Therefore,we sought to stratify our study population by age group to generate risk-based likelihoods that could be used to determine whether or not to perform ECC.

We found that ECC should be performed in females older than 50 years of age and when the cervix is more difficult to visualize.We know that hormone levels generally decrease with increasing age and that the endocervical canal may become more susceptible to development of lesions.This may explain why colposcopists prefer to perform endocervical sampling in older females but also suggests that adequate visualization of the surface of the cervix does not necessarily raise concern about precancerous lesions in the endocervix in younger females.Unfortunately,the evidence we found for identifying females most in need of ECC based on type of TZ was inconclusive.Previous studies have suggested that even when the TZ is completely visible with a colposcope,the findings on ECC may still be positive (22).Therefore,incomplete visualization of the TZ is more likely to be linked to the experience and skills of the colposcopist.The TZ is a particularly interesting aspect of colposcopy that must addressed to improve the accuracy of diagnosis (23)but does not necessarily raise concern about missed endocervical lesions;for that,we would need to use other indices in conjunction with more established indicators.In view of the significant predictors identified in this study,it seems important to be able to identify higher-risk patients to avoid unnecessary ECC,although there is still scope to reconsider risk thresholds with an element of flexibility,and lifestyle factors may need to be examined in more detail.

Multivariate analysis can obtain coefficients for each risk factor and then calculate specific risk values through the model’s formula.However,it can be difficult to integrate the predictive values for each of these indicators.Recently,Liet al.developed a predictive model for clinical decisionmaking regarding ECC and found that access to the requisite elements for significant clinical variables was linked to the ECC positivity rate (24).However,to determine which females required ECC was based on whether the patient had LSIL or worse rather than on HSIL+.We know that presence of cancer is more likely with HSIL+than with LSIL+(25).Furthermore,used of models based on inappropriate classification is likely to be futile.Another limitation of the study reported by Liet al.was that their prediction model was not externally validated,which means it has yet to be properly tested and therefore may not be as reliable as first thought.Our present study is an extension of their research and attempts to fill this knowledge gap.We hope that external validation of our novel multivariable predictive nomogram will allow the ECC HSIL+risk to be calculated on an individual basis for patients who are referred for colposcopy.

Our nomogram appears to have good discrimination ability and calibration for identifying those at high-risk of being ECC HSIL+and has been internally and externally validated.Our findings are further supported by the results of decision curve analysis,which suggest an increased net benefit from using this nomogram.The predictive items can be assessed in a colposcopy clinic,which adds to the ease of use.Therefore,our nomogram could be used in clinical practice.Clinicians may choose a wait-and-watch approach for females in whom the risk is estimated to be low and refer those at high risk for ECC to rule out occult HSIL+.However,there may be opportunities to develop this model further.More balanced cut-off values might be needed as a trade-off to limit the risk of a decrease in sensitivity if an excessive number of ECC procedures are performed.Another way of improving specificity while retaining high sensitivity might be to include novel predictors,such as E6/E7 mRNA or E6 oncoproteins.We deliberately did not classify risk as low,moderate or high,preferring instead to show defined sensitivity and specificity for each probability as a cut-off value for identifying individuals at high risk of ECC HSIL+.We chose this approach because we believe clinicians are better informed by calculating risk estimates for each individual patient and making decisions based on real-world conditions.Further research is needed to determine if there is any variation in cut-off values according to ethnicity.

Our study had several strengths and some limitations.First,our risk prediction model is based on a comprehensive set of parameters that are available in daily colposcopy practice.Second,unlike previous studies,we recognized the need for a large sample of hospitals in a country like China and used data from four centers to develop the nomogram and two independent external datasets for validation and assessment of its generalizability.Although ECC is not performed during colposcopy in all females,we had a rare opportunity to collect ECC data from thousands of colposcopic examinations performed at six centers in mainland China.Recent research suggests that artificial intelligence-guided colposcopy could have excellent performance (26-28).However,development of artificial intelligence imposes demands in terms of data collection.Therefore,we need to consider the data we collect and how we can network data across countries so that researchers can access huge datasets to further develop this prototype nomogram.

We are currently trying to link our ECC nomogram with an artificial intelligence-based colposcopy system to improve the detection of HSIL+and further standardize the entire diagnostic process in colposcopy clinics.However,it takes some time to enroll hospitals into research projects of this nature and there is a reluctance to do so,mainly because of concerns about patient privacy.Therefore,although this study had a number of advantages,we can only offer tentative recommendations because it focused specifically on females in mainland China.Other limitations concern data availability,in that we could not consider the additional diagnostic utility of ECC for detection of HSIL+that is missed by biopsy alone.Furthermore,it is not clear whether patients referred for colposcopy should undergo cervical biopsy only or whether they should be investigated by ECC,which is more invasive.Further research is needed to determine if unnecessary colposcopy examinations for detection of HSIL+can be avoided by performing biopsy and ECC at the same time.Finally,our results were based on diagnoses made by senior clinicians.More prospective clinical research is required to ensure that this nomogram can be used easily by less experienced clinicians and to validate the effectiveness of model-assisted clinicians,generally.

Conclusions

We developed and validated a predictive model by incorporating multiple clinically relevant variables to improve identification of ECC HSIL+cases in patients referred for colposcopy. This nomogram may help clinicians when making decisions about whether to perform ECC,thereby reducing the risk of overuse and minimizing the physical and psychological harm to females.Further prospective research at a global level is required before this tool can be incorporated into clinical practice.

Acknowledgements

This study was supported by CAMS Innovation Fund for Medical Sciences (No.CAMS 2021-I2M-1-004).

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.