Online calculator for predicting the risk of malignancy in patients with pancreatic cystic neoplasms: A multicenter, retrospective study

Dong Jiang, Zi-Xiang Chen, Fu-Xiao Ma, Yu-Yong Gong, Tian Pu, Jiang-Ming Chen, Xue-Qian Liu, Yi-Jun Zhao, Kun Xie, Hui Hou, Cheng Wang, Xiao-Ping Geng, Fu-Bao Liu

Abstract

Key Words: Pancreatic cystic neoplasms; Risk of malignancy; Nomogram; Model; Prediction; Calculator

lNTRODUCTlON

Pancreatic cystic neoplasms (PCNs) are estimated to be present in 2%-45% of the general population[1-3]. With rapid advancements in medical examination technologies over the past decades, the use of these technologies in the identification of PCNs has dramatically increased. Moreover, PCNs are frequently discovered in asymptomatic patients[4]. Consequently, centers with high volumes of patients undergoing pancreatic surgery are evaluating an increasing number of PCNs cases annually. Nevertheless, how to properly manage patients with PCNs remains a topic of debate.

PCNs are usually divided into serous cystic neoplasms (SCNs), mucinous cystic neoplasms (MCNs),intraductal papillary mucinous neoplasms (IPMNs) and other rare cystic lesions, such as solid pseudopapillary tumors (SPTs) and cystic neuroendocrine tumors (cNET) which show biological behavior ranging from benign to malignant[5,6]. Moreover, the main controversy regarding the treatment of PCNs is related to the inability to precisely determine their histopathological diagnosis without surgical resection[7,8], thereby precluding the accurate identification of different subtypes of PCNs. Consequently, clinicians are challenged to maintain a difficult balance between avoiding excessive surgical treatment and keeping a malignant lesion under surveillance.

Although previous studies have focused on the identification of different types of PCNs[9-11], only a few of these studies have emphasized the judgment or prediction of malignancy risk in PCNs. Thus,identifying a method that can accurately and preoperatively predict the risk of malignancy in PCNs is of great significance. Nomograms are effective statistical tools that enable the simultaneous consideration of various factors for clinicians to visualize the malignant probability of a neoplasm, allowing the formulation of an optimal therapeutic schedule. Nomogram possess other advantages including ease of comprehension, user-friendliness and personalized evaluation[12]. Therefore, this study aimed to establish a novel clinical online nomogram-based calculator to predict the risk of malignancy in PCNs.

MATERlALS AND METHODS

The clinical and pathological data of patients with PCNs who underwent surgery at the Department of General Surgery in three medical centers between January 2015 and December 2021 were collected and analyzed. Patients from the First Affiliated Hospital of Anhui Medical University and Second Affiliated Hospital of Anhui Medical University were set as the training cohort, while those from the First Affiliated Hospital of the University of Science and Technology of China were set as the validation cohort. All hospitals were high-volume surgical centers that employed similar therapeutic approaches for PCNs. Pathological diagnosis was confirmed by postoperative specimen examinations by two experienced pathologists. According to the histological classification of PCNs proposed by the World Health Organization (2019) and the Baltimore consensus meeting, PCNs are classified as those with lowor intermediate-grade dysplasia and those with high-grade dysplasia or invasive carcinoma[13,14].Patients from the three centers who met the following criteria were included: (1) Patients who underwent surgical treatment and were confirmed as showing SCNs, IPMNs or MCNs in postoperative pathological examinations; (2) Patients with complete clinical data, including medical history and laboratory test data; and (3) Patients who underwent preoperative imaging examinations. The following patients were excluded: (1) Patients who were pathologically confirmed as having other rare cystic lesions such as SPTs and cNET; (2) Patients who were pathologically confirmed as having other uncommon and undefined cystic tumors of the pancreas; and (3) Patients with a history of pancreatic surgery, radiotherapy or chemotherapy. All clinical data were screened and collected in a computerized database by a professional research assistant. In this study, patients were categorized as showing lowrisk (low- or intermediate-grade dysplasia) or high-risk (high-grade dysplasia or invasive carcinoma)disease based on the pathological diagnosis. The appearance of high-risk disease was characterized as a study endpoint. Figure 1 depicts a flowchart of patient enrollment and the inclusion process. This study was conducted in accordance with the Declaration of Helsinki and was approved by the institutional ethics committee (Quick-PJ2022-06-26). All included patients or their relatives provided written informed consent before their data were analyzed.

Perioperative management

In accordance with the similar preoperative evaluation protocol at all centers, all patients were routinely examined using blood tests, including routine blood counts and analysis of blood biochemistry,hemostatic function, immunological markers and tumor markers. Moreover, all patients underwent at least two preoperative imaging examinations through ultrasound (US), computed tomography (CT),magnetic resonance imaging or 2-18F-fluoro-2-deoxy-d-glucose positron emission tomography-CT,which provided information about the size, number, location and internal condition of the lesions.Definitive planning of the procedure was performed according to the examination results and the patient’s age, comorbidities and preferences.

Surgical procedures, including pancreatoduodenectomy, total pancreatectomy, partial pancreatectomy and distal pancreatectomy with or without splenic preservation, were performed depending on the tumor location and characteristics. Experienced pancreatic surgeons performed all surgical procedures. Meanwhile, the balance between the benefits of oncological clearance and the risks of inadequate parenchymal preservation was carefully considered.

Normative and meticulous postoperative care including monitoring of vital signs, maintenance of moderate tissue perfusion and support for nutritional needs was implemented for every patient at the initial stage.

Statistical analysis

All data, including demographic data, preoperative imaging data and clinical and pathological data were recorded in detail using a special recorder. Continuous variables were expressed as mean ± SD for normally distributed variables or median (interquartile range) for non-normally distributed variables and statistical tests were used as appropriate (the independent sample t-test or the Mann-Whitney U test). Categorical variables were expressed as numbers (n) and proportions (%) and compared appropriatelyviaFisher’s exact test or the chi-squared test. The cutoff values of continuous parameters were derived from the Youden index[15]. Univariable logistic regression analysis was performed on the training cohort to screen for risk factors associated with malignancy. On the basis of the univariable analysis, multivariable logistic regression analysis was conducted using variables that had statistical significance. A nomogram for the prediction of malignancy in patients with PCNs was developed based on the multivariable logistic regression model. The performance of the nomogram was evaluated using a concordance index (C-index) and calibration plots based on bootstrap validation method (n= 1000) to reduce overfitting bias. The area under the curve (AUC) of the receiver operating characteristic (ROC)curves and the quality indices such as sensitivity, specificity, accuracy, positive predictive value and negative predictive value with the associated 95% confidence intervals (CIs) of the model applied to the training and external validation cohorts were obtained. This allowed assessment of the predictive efficiency of the model in comparison with those of the risk factors identified in the relevant guidelines.The clinical usefulness of the nomogram was examined by determining the net benefit using decision curve analysis (DCA)[16]. DCA can assess the utility of models for decision making by plotting net benefit at a range of clinically reasonable risk thresholds. The predictive accuracy and clinical usefulness of the nomogram were demonstrated by clinical impact curves. Based on the sensitivity, specificity and predictive values, the optimal cutoff value was evaluated for accuracy. An online calculator for public use was created through shinyapps.io by RStudio to facilitate the use of the model in clinical practice.Statistical analyses were performed using R version 4.0.5 (http://www.r-project.org/, R Development Core Team) and SPSS version 23.0 (SPSS Inc, Armonk, NY, United States. In this study, the packages rms, nomogramFormula, DynNom and rmda were utilized for statistical analysis and figure drawing. A two-tailedPvalue ＜ 0.05 was considered significant.

Figure 1 Flowchart of patient recruitment and diagnosis. AHMU: Anhui Medical University; USTC: University of Science and Technology of China.

RESULTS

Patient cohorts and clinicopathologic features

The clinicopathologic data of 1764 consecutive patients who underwent pancreatic surgery at the First Affiliated Hospital of Anhui Medical University and the Second Affiliated Hospital of Anhui Medical University between January 2015 and December 2021 were collected. Among them, 246 patients (13.9%)who were pathologically confirmed as having IPMNs, MCNs or SCNs were classified as the initial target population. Of these 246 patients, 59 (24.0%) who did not fulfill the inclusion criteria were excluded, including 19 patients with incomplete clinical or imaging data, nine with a history of pancreatic surgery, 25 with mixed pathological characteristics and 6 with a history of radiotherapy or chemotherapy. Ultimately, 187 patients (76.0%) were included in the training cohort. Based on the same criteria, an independent group consisting of 103 patients from the First Affiliated Hospital of University of Science and Technology of China who were eligible for inclusion in the same period was included in the present study and served as an external validation cohort. The two cohorts showed no significant differences in baseline characteristics (P＞ 0.05; Tables 1 and 2).

Univariable and multivariable analysis in the training cohort

The results of the univariable and multivariable analyses of variables concerning malignancy in the training cohort were listed in Table 3. The optimal cutoff values of the neutrophil-to-lymphocyte ratio(NLR) was 2.288. Factors that significantly affected the risk of malignancy in the univariable analysis were then included in the multivariable analysis, which demonstrated that the presence of enhanced mural nodules [odds ratio (OR): 4.314; 95%CI: 1.618-11.503,P= 0.003], tumor diameter ≥ 40 mm (OR:3.514; 95%CI: 1.138-10.849,P= 0.029), main pancreatic duct dilatation (OR: 3.267; 95%CI: 1.230-8.678,P= 0.018), preoperative NLR ≥ 2.288 (OR: 2.702; 95%CI: 1.008-7.244,P= 0.048) and preoperative serum CA19-9 concentration ≥ 34 U/mL (OR: 3.267; 95%CI: 1.274-13.007,P= 0.018) were independent risk factors for a high risk of malignancy in patients with PCNs.

Development and evaluation of the predictive model

The factors that were proven to be related to a high risk of malignancy were utilized to establish a nomogram for evaluating the probability of malignancy (Figure 2). The model obtained using this approach showed good predictive capacity for estimating the risk of malignancy in patients with PCNs(training cohort: C-index, 0.824; 95%CI: 0.735-0.914,vsvalidation cohort: C-index, 0.893; 95%CI:0.823-0.963). The calibration plots demonstrated good consistency between the observed and predicted probabilities in both the internal and external validations. Both predicted and reference curves were approximately aligned which indicates good performance of the nomogram (Figure 3). Overall performance was scored at 0.091 using Brier's score which measured the difference between observedand predicted values (values closer to 0 indicate better predictive ability). A calibration slope of 1.0 was obtained to assess the agreement between observed and predicted values (values closer to 1.0 indicate better performance)[17]. An online calculator for predicting the risk of malignancy was created and was freely available at https://ahmuptt.shinyapps.io/JDYX/.

Table 1 Preoperative clinical characteristics of patients with pancreatic cystic neoplasms

Table 2 Preoperative imaging characteristics of patients with pancreatic cystic neoplasms

Comparison of the performance of the nomogram and the risk factors identified in the relevant guidelines

Using ROC analyses, the diagnostic performance of the nomogram established in this study was compared with those of the risk factors identified in the relevant guidelines[6,18], including tumor diameter ≥ 40 mm, enhanced mural nodules and main pancreatic duct dilatation (Figure 4). In the training cohort, the AUC values and 95%CIs of the nomogram and the three factors (tumor diameter ≥40 mm, enhanced mural nodules and main pancreatic duct dilatation) were 0.824 (0.735-0.914), 0.619(0.496-0.742), 0.692 (0.575-0.810) and 0.653 (0.540-0.766), respectively. In the validation cohort, these values were 0.893 (0.823-0.963), 0.718 (0.560-0.876), 0.665 (0.506-0.824) and 0.672 (0.524-0.820),respectively. In both cohorts, significant statistical differences were found between the nomogram and the three factors (P＜ 0.05). The training cohort had an accuracy, sensitivity, specificity, positive predictive value and negative predictive value of 0.829, 0.643, 0.862, 0.451 and 0.932, respectively. In the validation cohort, these values were 0.925, 0.882, 0.826, 0.501 and 0.973, respectively. Thus, thenomogram showed high accuracy in predicting the risk of malignancy in patients with PCNs. When considering the maximum Youden index value, the optimal cutoff value of the nomogram was 160.8 in the ROC curve and the sensitivity and specificity for differentiating between high and low risk were 64.3% and 86.2%, respectively. This cut off value is used to categorize patients with total nomogram score of ＜ 160.8 points or ≥ 160.8 points as either low- or high-risk group, respectively. In the training cohort, DCA showed that using the nomogram to predict the risk of malignancy had a greater net benefit than the other three factors when the threshold probability ranged from 0.2 to 1.0. In the validation cohort, DCA showed that the nomogram had a greater net benefit when the threshold probability ranged from 0.0 to 0.4 (Figure 5). Figure 6 depicts the number of high-risk patients (solid red line) and the number of high-risk patients with the outcome (black dotted line) at different threshold probabilities within a given population. In both the training and validation cohorts, the solid red and black dotted lines show a great fit indicating that the model has remarkable predictive ability.

Table 3 Univariable and multivariable logistic regression analyses of the risk factors for high risk of malignancy in patients with pancreatic cystic neoplasms

DlSCUSSlON

The increasing use of high-resolution cross-sectional imaging techniques has resulted in more frequent detection of PCNs in recent years[19]. However, accurate prediction of the risk of malignancy of PCNs as early as possible is of great value in developing more appropriate individualized treatment strategies.In the current study, we developed and validated a preoperative clinical model that strongly predicted the risk of high-grade dysplasia or invasive cancer in patients with PCNs.

Figure 2 Nomogram for predicting risk of malignancy in the training cohort. NLR: Neutrophil-to-lymphocyte ratio.

Figure 3 Calibration curves for predicting the risk of malignancy. The nomogram had concordance index values of 0.824 and 0.893 in the training and validation cohort, respectively. A: Training cohort; B: Validation cohort.

As shown in Tables 1 and 2, our study analyzed dozens of objective factors, and the findings indicated that for patients with PCNs, lesions with enhanced mural nodules, tumor diameter ≥ 40 mm,main pancreatic duct dilatation, preoperative NLR ≥ 2.288 and preoperative serum CA19-9 levels ≥ 34 U/mL were significant independent predictors. These factors were combined with patients’ preoperative imaging and serological data to quantify the risk of malignancy concisely and spontaneously.The sufficient statistical power of the model for predicting the risk of malignancy was verified based on multiple validation methods. The C-index of the model in the validation cohort was 0.893 (0.823-0.963),highlighting the ability of the model to distinguish low-risk disease from high-risk disease in a large group of patients. In addition, we used an independent validation dataset that effectively decreased the risk of overfitting the model to an individual dataset. The similarity between the C-index of the training[0.824 (0.735-0.914)] and validation [0.893 (0.823-0.963)] cohorts suggested that this model was widely applicable. In addition, a mobile-friendly online version of the nomogram was programmed to eliminate the inconvenience of traditional nomograms in clinical use.

Figure 4 Receiver operating characteristic curve of the nomogram and other three risk factors identified in guidelines. A: Training cohort; B:Validation cohort. AUC: Area under the curve.

The latest guidelines, including the European evidence-based guidelines on PCNs and the International Association of Pancreatology guidelines of 2017, have described the management strategies for different subtypes of PCNs in detail[6,18]. In addition, many published nomogram models for predicting malignancy in patients with a specific subtype of PCNs, such as IPMNs, have been established to facilitate the preoperative prediction of malignant lesions[20-22]. Nevertheless, real-world preoperative diagnosis of the different PCN subtypes remains difficult due to their similarities in both clinical and imaging features. In particular, accurate diagnosis of pathological neoplasms and preoperative assessments of the benign or malignant status of some lesions with atypical manifestations are even more difficult[23]. Previous studies have reported differing probabilities for the accurate preoperative identification of each subtype. Salviaet al[24] found that in 476 patients with PCNs, 78%showed postoperative pathological examination results consistent with the preoperative diagnosis[24].Furthermore, a multicenter study of 2251 patients in China showed that the preoperative diagnosis was not always accurate for a specific subtype with a correct diagnosis rate of only 33%[25]. Meanwhile, the risk factors proposed to be associated with malignant lesions in the guidelines lack universality,efficiency and accuracy in clinical applications to a certain extent. In this study, a model with strong practicability and high predictive efficiency was constructed by analyzing PCNs showing similar clinical and imaging manifestations across subtypes. Therefore, this model can help clinicians predict the risk of lesion malignancy when they are unable to accurately determine the PCN subtype or have difficulties in confirming the nature of the tumor.

NLR was also included as a laboratory indicator in the nomogram. Researches have shown that inflammation contributes to the occurrence and progression of tumors by releasing cytokines and other inflammatory mediators[26,27]. NLR measurement can serve as a simple and convenient way to assess the systemic inflammatory response and many recent studies have confirmed that inflammatory markers play an important role in predicting benign and malignant PCNs[28-30]. In the current study, a significant correlation was found between NLR and high-risk groups by multivariable analysis and the cutoff value of NLR was calculated as 2.43 which was rarely studied and reported in previous studies.We believe that neutrophils can recruit and activate inflammatory cells by producing cytokines and chemokines thus acting on the tumor microenvironment. In addition, reactive oxygen species and proteases produced by neutrophils can regulate tumor cell proliferation, angiogenesis and metastasis[31]. Thus, an increase in NLR may indicate a macroscopic withdrawal of inflammatory and immune responses in the local tumor microenvironment.

The nomogram was chosen to establish the model mainly because of its ability to assign risk probabilities on a continuous scale as an individualized risk score, rather than simply dividing patients into malignant and benign groups. This allows for additional risk stratification and may help patients and doctors tailor treatment decisions based on patients’ individual risks. In addition, as shown in Figure 2,the nomogram showed that the model score for cases with serum CA19-9 levels ≥ 34 U/mL (100 points)was significantly higher than that for cases with an NLR of ≥ 2.288 (61 points), suggesting that serum CA19-9 levels ≥ 34 U/mL increase the possibility of a malignant lesion significantly. Moreover, the additive effect of several risk factors is important clinically. In this context, the patient-specific quantitative estimation of risk is valuable. Therefore, in cases where benign and malignant PCNs cannot be easily distinguished, clinicians can calculate the total nomogram scores using this predictive model.We propose that surgical treatment should be considered as the first choice for patients with a total nomogram score ≥ 160.

Figure 5 Decision curve analysis of the nomogram and other three risk factors identified in guidelines. The x-axis represents the threshold probability and the y-axis represents the net benefit. In the training cohort, the nomogram adds more net benefit than the other three factors when the threshold probability ranged from 0.2 to 1.0. In the validation cohort, the nomogram adds more net benefit than the other three factors when the threshold probability ranged from 0.0 to 0.4. A: In the training cohort; B: In the validation cohort.

In recent years, machine learning has rapidly developed as a tool with promising results and improved usability[32]. Machine learning addresses many preexisting and novel medical challenges and is widely discussed by researchers and practitioners alike. Machine learning algorithms have many advantages, such as improved flexibility and scalability when compared with traditional statistical methods. This makes machine learning advantageous for many tasks, including risk stratification,diagnosis, classification and survival predictions. Another advantage of machine learning is the ability to analyze diverse data types (e.g., demographic data, laboratory findings, imaging data and doctors'free-text notes) and to incorporate them into predictions for disease risk, diagnosis, prognosis and appropriate treatment[33]. Despite these advantages, the application of machine learning in healthcare delivery also presents unique challenges, including the need for data pre-processing, model training and refinement of the system with respect to the actual clinical problem. Also crucial are ethical considerations which include medico-legal implications, doctors' understanding of machine learning tools, and data privacy and security[34]. In summary, attempts can be made to refine the model further using advanced machine learning in the future.

The present study had several limitations that merit discussion. First, endoscopic US (EUS) helps identify PCNs with features that may indicate the need for surgical resection[6]. However, in our study,only less than 10% of the patients underwent EUS. This may be attributable to the considerable interobserver variation in EUS-based diagnoses. On the other hand, data on EUS-based differentiation between benign and malignant PCNs are inconsistent[35,36]. Moreover, economic affordability is an important consideration, especially for most patients from rural areas. Second, heterogeneity in pathological diagnosis to determine the grade of dysplasia or malignancy and heterogeneity between different imaging examinations may exist. Third, this was a retrospective study considering only patients who underwent surgery which meant that it had inherent limitations resulting from potential selection biases; prospective validation is therefore required to confirm the value of the findings.

Figure 6 Clinical impact curves of the nomogram. At different threshold probabilities within a given population, the number of high-risk patients (solid red line) and the number of high-risk patients with the outcome (black dotted line) are shown. In both training and validation cohort, the solid red line and black dotted line show a great fit. A: In the training cohort; B: In the validation cohort.

CONCLUSlON

In conclusion, we developed and validated a novel online calculator using a nomogram based on widely available data to predict the risk of malignancy in patients with PCNs dynamically. The calculator is user-friendly, highly accurate and well validated. Clinicians can use it to alert patients at high risk of malignancy at early stages and to design individual therapy for them.

ARTlCLE HlGHLlGHTS

Research background

Efficient and practical methods for predicting the risk of malignancy in patients with pancreatic cystic neoplasms (PCNs) are currently lacking.

Research motivation

Currently, there is no effective clinical prediction model for patients with PCNs and no large study has been conducted to predict malignant risk.

Research objectives

The aim of this study was to identify the risk factors influencing the malignant risk of PCNs and develop a prediction model that is useful for clinical surgeons when making decisions regarding surgical interventions.

Research methods

Data collected in three major medical centers were analyzed to identify independent risk parameters and propose a calculator for patients with PCNs. A number of statistical indices, such as concordance index, calibration curves, area under the curve, decision curve analysis, CIC and others were used to evaluate the performance of the nomogram.

Research results

Five factors, including enhanced mural nodules, tumor diameter ≥ 40 mm, main pancreatic duct dilatation, preoperative neutrophil-to-lymphocyte ratio ≥ 2.288 and preoperative serum CA19-9 concentration ≥ 34 U/mL were found to independently influencing the risk of malignancy. As a result, the model we constructed has a greater predictive value than the factors identified in relevant guidelines.

Research conclusions

For the first time, a model was developed to predict the malignant risk of PCNs and an online calculator was further established to guide decision-making.

Research perspectives

More medical centers included, more data collection and application of “Artificial Intelligence”.

ACKNOWLEDGEMENTS

We would like to thank Prof. Faming Pan (Department of Epidemiology and Biostatistics, School of Public Health, Anhui Medical University), who has made valuable support to the data quality and data analysis which ensured the reliability of the study.

FOOTNOTES

Author contributions:Jiang D, Chen JM, Chen ZX and Ma FX contributed to the data analysis and participated in drafting the article; Liu XQ, Gong YY, and Pu T extracted the clinical data and calculated the clinical correlations; Xie K and Zhao YJ interpreted the results and revised the manuscript; Liu FB, Hou H, Wang C and Geng XP gave final approval of the version to be published; All authors contributed to the design and interpretation of the study and to further drafts and approved the final version to be published.

Supported byUniversity Natural Science Research Project of Anhui Province, No. KJ2021ZD0021.

lnstitutional review board statement:The study was reviewed and approved by the Institutional ethics committees of the First Affiliated Hospital of Anhui Medical University (Approval Quick-PJ2022-06-26).

lnformed consent statement:All study participants or their legal guardian provided informed written consent prior to study enrollment.

Conflict-of-interest statement:All the authors report no relevant conflicts of interest for this article.

Data sharing statement:No additional data are available.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:China

ORClD number:Dong Jiang 0000-0003-3372-6062; Tian Pu 0000-0002-6961-0814; Jiang-Ming Chen 0000-0003-4683-5866;Yi-Jun Zhao 0000-0001-9082-8861; Kun Xie 0000-0002-2491-4988; Hui Hou 0000-0002-5997-1229; Cheng Wang 0000-0001-7899-2079; Xiao-Ping Geng 0000-0002-6219-6366; Fu-Bao Liu 0000-0003-3431-2187.

S-Editor:Fan JR

L-Editor:Filipodia

P-Editor:Yu HG