Application of an artificial intelligence system for endoscopic diagnosis of superficial esophageal squamous cell carcinoma

Qian-Qian Meng, Ye Gao, Han Lin, Tian-Jiao Wang, Yan-Rong Zhang, Jian Feng, Zhao-Shen Li, Lei Xin, Luo-Wei Wang

Abstract

Key Words: Computer-aided diagnosis; Artificial intelligence; Deep learning; Esophageal squamous cell carcinoma; Early detection of cancer; Upper gastrointestinal endoscopy

lNTRODUCTlON

Esophageal cancer is the seventh most common and sixth most fatal malignancy[1], and esophageal squamous cell carcinoma (ESCC) is the most prevalent histological type[2]. Screening high-risk populations using upper gastrointestinal endoscopy may facilitate early detection and treatment[3], and markedly improve the prognosis of ESCC[4]. However, due to its subtlety and flat appearance under white light imaging (WLI) or image-enhanced endoscopy, identification of early stage ESCC may be especially difficult for inexperienced endoscopists[5-7]. A recent multicenter cohort study found that 6.4% of all esophageal cancers were missed during upper gastrointestinal endoscopy, which was associated with poor survival[8].

In recent years, artificial intelligence (AI)-aided endoscopy has garnered attention, and several studies have heralded this technique as a promising tool for improving the detection of early ESCC[9-12]. A systematic review showed that the pooled sensitivity and specificity of AI for ESCC diagnosis were 0.95 and 0.92, respectively, and AI performed better than endoscopists, although the differences were not statistically significant[13]. However, few computer-assisted diagnosis (CAD) systems for ESCC that support WLI and narrow-band imaging (NBI) have been applied in clinical practice. We developed and validated a novel CAD system with a deep neural network algorithm to detect superficial ESCC using upper endoscopy with WLI and NBI. Moreover, we assessed its value for endoscopists with differing experience levels of superficial ESCC detection and investigated the characteristics of lesions that were inaccurately identified by the CAD system.

MATERlALS AND METHODS

Preparation of datasets for training and testing

Static and non-magnified WLI or NBI images used for CAD system training and testing for superficial ESCC were retrospectively retrieved from endoscopic databases of four general hospitals in mainland China between January 2016 and April 2019. For the training dataset, 1503 WLI images and 2100 NBI images from 622 cases of superficial ESCC and high-grade intraepithelial neoplasia (HGIN), and 523 WLI images and 321 NBI images from 215 non-cancerous cases (including normal, esophagitis,submucosal lesion, and leukoplakia) were obtained from the First Affiliated Hospital of Zhengzhou University, Xingtai First Hospital, and Tangyin People's Hospital. For the independent testing dataset,577 WLI images and 793 NBI images from 209 cases of superficial ESCC and HGIN, and 209 WLI images and 116 NBI images from 114 non-cancerous cases were obtained from Changhai Hospital. All images were captured using an Olympus gastroscope (GIF-H260, GIF-H260Z, GIF-H290, GIF-Q260J, or GIFHQ290; Olympus, Tokyo, Japan). Poor-quality images, including blurred images, and those with mucus or foam that prevented adequate mucosal inspection were excluded from the study. The personal data of the patients in the images were hidden. This study was approved by the ethics committee of Shanghai Changhai Hospital (No. CHEC2019-002).

All superficial ESCC and HGIN images were obtained from patients who underwent subsequent endoscopic submucosal dissection. Additionally, histologic results including margin status and invasion depth of endoscopic submucosal dissection specimens were retrieved. Three experienced gastrointestinal (GI) pathologists conducted histological assessments. Histological images were first reviewed by two independent pathologists who determined the final pathological result through diagnostic consensus. In case of disagreement, images were reviewed by a third pathologist, and the final decision was discussed. Non-cancerous images were confirmed by three experienced endoscopists (all with ＞ 15 years of experience), before inclusion in training and testing datasets.

All images were annotated by the three endoscopists mentioned above. The first endoscopist reviewed and marked the WLI and NBI images (margins of the lesion) using specific frames. The second endoscopist reviewed the annotations of the first, and if in agreement, the annotation was completed.Alternately, the images were reviewed by a third endoscopist, and the final decision was made in unison. Paris classification subtypes of lesions were also recorded[14].

Development of the CAD system

CAD system development was based on the YOLO v5 deep-learning algorithm. YOLO v5 employs cross stage partial network as the backbone for feature extraction and path aggregation network to generate a feature pyramid network to perform feature aggregation and pass it to the prediction mode. YOLO v5 demonstrated increased efficiency and efficacy compared to previous versions. To improve network accuracy, we modified the loss function of YOLO V5 and introduced the mapping distance of the aspect ratio. The input of this network was esophageal images, and the output was an indicator frame that captured the rectangular region of interest, together with the predicted quantitative level of confidence for ESCC above the frame (Figure 1).

Testing of the CAD system

The diagnostic accuracy of the CAD system was validated using a previously established independent testing dataset. To compare the performance of the CAD system with that of endoscopists and investigate its added value, four endoscopists from Changhai Hospital were invited to review the images in the testing dataset, independently and with the assistance from the CAD system. Two experts(endoscopy experience ≥ 10 years) and two non-expert endoscopists (endoscopy experience ＞ 1 year but＜ 5 years) participated. None were involved in the selection and annotation of images. All images in the testing dataset, including 598 WLI and 817 NBI cancerous and 209 WLI and 116 NBI non-cancerous images, were randomly sequenced, and all participating endoscopists independently reviewed the images and made diagnoses within one day. After a washout period of one month, the images were randomly re-sequenced, and endoscopists reviewed the images using the results of the CAD system as a reference. Throughout the study, participating endoscopists were unaware of the correct endoscopic and histologic diagnoses, their own performance, and the diagnostic accuracy of the CAD system.

Outcome measures

Primary outcome measures included the area under the receiver operating characteristic curve(AUROC), accuracy, sensitivity, and specificity of the CAD system. Secondary outcome measures included positive predictive value (PPV) and negative predictive value (NPV) of the CAD system;diagnostic performance of participating endoscopists; and improvement in diagnostic accuracy,sensitivity, and specificity in the expert and non-expert groups, after referring to the CAD system results.

Statistical analysis

AUROC, accuracy, sensitivity, specificity, PPV, and NPV were calculated, and the binominal 95%confidence intervals (CI) were estimated using the CAD-predicted confidence ≥ 0.5, as the prespecified criteria for testing positive. The Delong test was used to compare different AUROCs. A two-sided McNemar test was used to compare differences in accuracy, sensitivity, and specificity. Differences in more than three groups were compared using Cochran’sQtest. Additionally, the McNemar test was used for post-hoc comparisons ofPvalues adjusted using Bonferroni’s method. Pearson’s chi-squared test,ttest, and Mann-WhitneyUtest were used wherever applicable. Superficial ESCC lesions with a CAD-predicted confidence ＜ 0.5 and an intersection over union (IoU) value ＜ 0.45 were defined as inaccurately identified. APvalue ＜ 0.05 was considered statistically significant. All statistical analyses were performed using SPSS 26.0 (SPSS Inc., Chicago, IL, United States). The statistical methods of this study were reviewed by XFY from Department of Health Statistics, Naval Medical University.

Figure 1 The architecture of the computer-assisted diagnosis system. The input of this network was the esophageal images, and the output was an indicator frame that captured the interest region, together with the predicted quantitative level of confidence for esophageal squamous cell carcinoma.

RESULTS

Characteristics of subjects and lesions in test dataset

Table 1 summarizes the characteristics of the patients and lesions in the testing dataset. For 209 cases of superficial ESCC, the lesion size was 2.0 (1.2-4.0) cm and most (82.3%) were located in the middle thoracic esophagus. Flat-type (0-IIb) lesions accounted for 53.6%, and most lesions involved ＜ 25% of the esophageal circumference. Regarding histologic depth of invasion, 165 Lesions (78.9%) were limited to the lamina propria (LPM), 38 (18.2%) invaded the muscularis mucosa or superficial submucosa, and 6(2.9%) extended beyond the superficial submucosa. Of the 114 non-cancerous cases, 23 were diagnosed with gastroesophageal reflux disease, five were diagnosed with esophageal submucosal lesions, and 86 were judged to be normal on upper gastrointestinal endoscopy.

Diagnostic performance of the CAD system

The receiver operating characteristic (ROC) curve of the CAD system is shown in Figure 2. The AUROC of the WLI, NBI, and combined modes were 0.970, 0.988, and 0.982, respectively. The AUROC of WLI,NBI, and the combined mode did not differ significantly. The diagnostic performance measures of the CAD system in the testing dataset are summarized in Table 2. In per-patient analysis, the accuracy,sensitivity, specificity, PPV, and NPV of the CAD system were 92.9%, 91.9%, 94.7%, 97.0%, and 86.4%,respectively. The accuracy (P＜ 0.001), sensitivity (P＜ 0.001), and specificity (P= 0.015) of the CAD system were significantly different among the different imaging modes. In post-hoc comparisons, the accuracy of the NBI (adjustedP＜ 0.001) and combined modes (adjustedP= 0.002) was significantly higher than that of the WLI mode. Furthermore, the sensitivity of NBI (adjustedP＜ 0.001) and combined modes (adjustedP= 0.014) was significantly higher than that of the WLI mode. No statistically significant differences in specificity were found in post-hoc comparisons between the two imaging modes. The results of the per-imaging analysis are presented in Table 2.

Comparison between the CAD system and endoscopists

Figure 3 compares the diagnostic performance of the CAD system under the combined WLI and NBI modes, among endoscopists with different experience levels. The accuracy (91.0%vs78.3%,P＜ 0.001),sensitivity (90.0%vs76.1%,P＜ 0.001), and specificity (93.0%vs82.5%,P= 0.002) were significantly higher for expert endoscopists than for non-expert endoscopists. The diagnostic performance of theCAD system was significantly higher than that of non-expert endoscopists (P＜ 0.001) and similar to that of expert endoscopists.

Table 1 Characteristics of subjects and lesions involved in the testing dataset

Table 2 Diagnostic performance of the computer-assisted diagnosis system in the testing dataset

Improvement after referring to CAD

Figure 4 depicts improved diagnostic performance of endoscopists for superficial ESCC after referring to the results of the CAD system. The accuracy (88.2%vs78.3%,P＜ 0.001), sensitivity (87.6%vs76.1%,P＜ 0.001), and specificity (89.5%vs82.5%,P= 0.039) of the non-expert endoscopists were all significantly improved after using the CAD system, whereas the diagnostic performances of expert endoscopists did not differ significantly between the two stages. After referring to the CAD system, the accuracy (88.2%vs93.2%,P＜ 0.001) and sensitivity (87.6%vs92.3%,P= 0.013) of the non-experts were still significantly lower than those of the experts, while the disparities were markedly decreased. The specificity (89.5%vs94.7%,P= 0.124) was similar between experts and non-experts after referring to the CAD.

Inaccurately identified lesions

Under the prespecified definition, 23 (11.0%) lesions were inaccurately identified. Table 3 compares the characteristics of lesions with accurate and inaccurate identifications. The proportion of type 0-IIb lesions was significantly higher in inaccurately identified lesions (73.9%vs51.1%,P= 0.038), and the proportion of lesions limited to the LPM was higher in the inaccurate identification group, although not significantly so (91.3%vs77.4%,P= 0.123).

Table 3 Characteristics of lesions that were inaccurately identified by the computer-assisted diagnosis system

Figure 2 Receiver operating characteristic curve for the computer-assisted diagnosis system in the testing dataset. CAD: Computer-assisted diagnosis; AUC: Area under the curve; WLI: White light imaging; NBI: Narrow-band imaging.

DlSCUSSlON

Figure 3 Comparison of diagnostic performances between the computer-assisted diagnosis system and endoscopists for superficial esophageal squamous cell carcinoma. A: Accuracy; B: Sensitivity; C: Specificity. aP ＜ 0.05; bP ＜ 0.01. CAD: Computer-assisted diagnosis.

Figure 4 The improvements in the diagnostic performance of endoscopists after referring to the results of computer-assisted diagnosis system. A: Accuracy; B: Sensitivity; C: Specificity. CAD: Computer-assisted diagnosis.

Upper gastrointestinal endoscopy is critical for screening and early diagnosis of ESCC. However, the subtle endoscopic appearance of most superficial ESCC types is challenging for early detection and relies greatly on the skills and experience of endoscopists. Herein, we developed and validated a novel deep-learning-based CAD system for the detection of superficial ESCC in the WLI and NBI modes of upper gastrointestinal endoscopy. The AUROC, accuracy, sensitivity, and specificity of the CAD system were 0.982, 92.9%, 91.9%, and 94.7%, respectively. The diagnostic performance was significantly superior to that of non-experts and comparable to that of expert endoscopists. After referring to the results of the CAD system, the detectability of non-expert endoscopists significantly improved, and disparities between experts and non-experts markedly decreased. This system may assist in improving the detectability of superficial ESCC, particularly for inexperienced endoscopists and in undeveloped areas with limited resources.

Horieet al[9] developed a deep learning-based CAD system for superficial ESCC, with a sensitivity of 98% and low PPV of 40%. Zhaoet al[15] developed an AI-based model for magnifying NBI images for automated classification of intrapapillary capillary loops, with a mean diagnostic accuracy of 89.6%. In a recent study by Guoet al[16], the authors developed a CAD system for real-time automated diagnosis of precancerous lesions and ESCC based on endoscopic video datasets, which achieved a per-lesion sensitivity of 100%. A systematic review by Visaggiet al[13] reported that the AUROC, sensitivity, and specificity of AI in diagnosing ESCC were 0.97, 0.95, and 0.92, respectively. However, few existing systems can support WLI and NBI, both of which are indispensable to the diagnosis of superficial ESCC under the same algorithm framework. We developed the current system under the updated YOLO v5 algorithm network supporting WLI and NBI, and achieved an AUROC, accuracy, sensitivity, and specificity of 0.982, 92.9%, 91.9%, and 94.7%, respectively, which was comparable to the best performance reported in previous studies. Esophageal endoscopic imaging is affected by factors such as esophageal peristalsis, contraction, and imaging quality. Therefore, to supplement data by mimicking the above influencing factors, we added undistorted imaging correction enhancement based on the original data enhancement method of YOLO v5.

Superficial ESCC is difficult to inspect under WLI; however, NBI may significantly improve detectability with similar sensitivity and higher specificity than Lugol chromoendoscopy[6]. This may explain the demonstrated decrease in the accuracy and sensitivity of the CAD system, without NBI. Caiet al[10]reported on a deep neural network (DNN)-based system that utilizes WLI with high accuracy and sensitivity, for early ESCC detection. They subsequently developed a CAD-NBI system and found that its accuracy (94.3%vs89.5%,P= 0.028) and specificity (96.7%vs83.1%,P＜ 0.001) were significantly better than those of the CAD-WLI system[12]. By using CAD-WLI and CAD-NBI, endoscopists can significantly improve their diagnostic efficacy. Taken together, we recommend the combined implementation of NBI with other image-enhancing techniques for the detection of superficial ESCC,even under the CAD system.

To assess the clinical application value of the CAD system, endoscopists of varying experience reviewed the entire testing dataset independently and after referring to the diagnostic results of the CAD system. During the independent review, non-expert endoscopists showed significantly lower accuracy, sensitivity, and specificity than the experts, which highlighted the difficulty of novices in detecting superficial ESCC and disparities in detectability among endoscopists. The diagnostic performance of the non-experts significantly improved after referring to the CAD system. This suggests that the results of the CAD system provide useful references for non-expert endoscopists and facilitate further inspection and evaluation of potentially malignant lesions. The improved diagnostic performance of non-experts remained unequal to that of the CAD system, suggesting that endoscopists had made independent decisions, rather than relying completely on CAD. Similar results were reported in previous studies[10,12,17]. In the current study, although the performance of non-experts was still inferior to that of the experts after referring to the CAD system, the disparity markedly decreased. This suggests that the CAD system can homogenize the performance of endoscopists by overcoming the differences among endoscopists with different experience levels. In clinical practice, the usefulness of high-definition and imaging-enhanced endoscopy may be jeopardized by the inadequate number of highly trained endoscopists. In contrast, the CAD system does not require additional training and has instead been found to improve the performance of non-expert endoscopists by approximating the level of expertise. Thus, the CAD system may be especially useful in resource-limited areas, to improve the quality of endoscopic services and patient outcomes.

To our knowledge, this is the first study to analyze the characteristics of inaccurately identified lesions. We defined cases of inaccurate identification based on the predicted confidence and accuracy of the location frame. Endoscopists should be aware that flat-type and earlier-stage lesions limited to the LPM may be associated with an increased risk of inaccurate identification. To avoid a missed diagnosis of superficial ESCC, we recommend the use of Lugol’s staining or magnifying endoscopy for flat-type lesions that are difficult to distinguish even under CAD.

In clinical practice, the CAD system automatically captures images as endoscopists inspect the esophagus, once integrated into an endoscopy workstation. Additionally, images frozen by endoscopists were input into the CAD system for analysis and detection of potential lesions. The CAD system then address potential malignant lesions with an indicator frame and a predicted quantitative level of confidence. Lesions with a predicted confidence ≥ 0.5 are strongly suggested for further evaluation and endoscopic biopsy. Other lesions may also be further evaluated or biopsied based on endoscopists’discretion. The diagnostic accuracy for ESCC is significantly improved after referring to the CAD system, particularly for non-expert endoscopists. Moreover, an updated CAD system that can process real-time videos or images with suboptimal quality is in development and may further facilitate clinical practice. In future, both endoscopists and patients may benefit from the increasing application of AI in endoscopy.

Our study had several limitations. First, the number of included endoscopists was relatively small,and the representation might be inadequate. Second, only still and non-magnified images were included; thus, the identification of videos and types of intrapapillary capillary loops could not be achieved. Third, the numbers and angles of lesions in NBI and WLI images were not identical in the testing dataset, and biases may exist in the comparison of diagnostic performance among the different modes of the CAD system.

CONCLUSlON

We developed and validated a deep-learning CAD system, for the endoscopic detection of superficial ESCC, that achieved a high diagnostic accuracy comparable to that of expert endoscopists and could significantly improve the detecting ability of non-expert endoscopists.

ARTlCLE HlGHLlGHTS

Research background

Esophageal squamous cell carcinoma (ESCC) is a leading cause of cancer-related morbidity and mortality worldwide. Upper gastrointestinal endoscopy is critical for ESCC detection; however,endoscopists require long-term training to avoid missing superficial lesions. Artificial intelligence (AI)has been increasingly investigated to assist endoscopic diagnosis.

Research motivation

AI has shown promising results for endoscopic diagnosis of superficial ESCC. However, few AI-based computer-assisted diagnosis (CAD) systems for ESCC that support white-light and narrow-band imaging have been applied in clinical practice.

Research objectives

We aimed to develop a CAD system for endoscopic detection of superficial ESCC and investigate its application value.

Research methods

We configured the CAD system for white-light and narrow-band imaging modes based on the YOLO v5 algorithm. A total of 4447 images from 837 patients and 1695 images from 323 patients were included in the training and testing datasets, respectively. Two experts and two non-expert endoscopists reviewed the testing dataset independently and with computer assistance. The diagnostic performance was evaluated in terms of the area under the receiver operating characteristic curve, accuracy, sensitivity,and specificity.

Research results

The area under the receiver operating characteristics curve, accuracy, sensitivity, and specificity of the CAD system were 0.982 [95% confidence interval (CI): 0.969-0.994], 92.9% (95%CI: 89.5%-95.2%), 91.9%(95%CI: 87.4%-94.9%), and 94.7% (95%CI: 89.0%-97.6%), respectively. The accuracy of CAD was significantly higher than that of non-expert endoscopists (78.3%,P＜ 0.001 compared with CAD) and comparable to that of expert endoscopists (91.0%,P= 0.129 compared with CAD). After referring to the CAD results, the accuracy of the non-expert endoscopists significantly improved (88.2%vs78.3%,P＜0.001).

Research conclusions

The diagnostic performance of the CAD system is promising and may assist in improving detectability,particularly for inexperienced endoscopists.

Research perspectives

An updated CAD system that can process real-time videos or images with suboptimal quality is in development. Randomized controlled trials are warranted to investigate the clinical pragmaticality.

FOOTNOTES

Author contributions:Wang LW was the guarantor and designed the study; Lin H, Wang TJ, Zhang YR, Feng J participated in the acquisition, analysis, and interpretation of the data; Gao Y, Meng QQ drafted the initial manuscript; Xin L, Li ZS revised the article critically for important intellectual content; Meng QQ, Gao Y and Lin H contributed equally to this article.

Supported byShanghai Science and Technology Innovation Action Program, No. 21Y31900100; and 234 Clinical Research Fund of Changhai Hospital, No. 2019YXK006.

lnstitutional review board statement:The study was reviewed and approved by the Shanghai Changhai Hospital Ethical Committee.

lnformed consent statement:All study participants, or their legal guardian, provided informed written consent prior to study enrollment.

Conflict-of-interest statement:Dr. Wang reports grants from The Science and Technology Commission of Shanghai Municipality during the conduct of the study. Other authors report no relevant conflicts of interest for this article.

Data sharing statement:No additional data are available.

STROBE statement:The authors have read the STROBE Statement-checklist of items, and the manuscript was prepared and revised according to the STROBE Statement-checklist of items.

Open-Access:This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers. It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is noncommercial. See: https://creativecommons.org/Licenses/by-nc/4.0/

Country/Territory of origin:China

ORClD number:Qian-Qian Meng 0000-0003-4719-1425; Ye Gao 0000-0002-6338-9602; Han Lin 0000-0002-0137-5176; Tian-Jiao Wang 0000-0002-7537-8089; Yan-Rong Zhang 0000-0002-5779-3828; Jian Feng 0000-0001-9022-462X; Zhao-Shen Li 0000-0002-1650-4014; Lei Xin 0000-0002-0495-8780; Luo-Wei Wang 0000-0002-6588-0542.

S-Editor:Gong ZM

L-Editor:A

P-Editor:Gong ZM