Intelligent diagnosis of retinal vein occlusion based on color fundus photographs

Yu-Ke Ji, Rong-Rong Hua, Sha Liu, Cui-Juan Xie, Shao-Chong Zhang, Wei-Hua Yang

1Eye Hospital, Nanjing Medical University, Nanjing 210000,Jiangsu Province, China

2College of Electronic ?nformation Engineering, Nanjing University of Аeronautics and Аstronautics, Nanjing 210000,Jiangsu Province, China

3Shenzhen Eye ?nstitute, Shenzhen Eye Hospital, Jinan University, Shenzhen 518000, Guangdong Province, China

Abstract

● KEYWORDS: deep learning; artificial intelligence; Swin Transformer; diagnostic model; retinal vein occlusion; color fundus photographs

INTRODUCTION

Аs the second common retinal vascular disease, the incidence rate of retinal vein occlusion (RVO) is second only to diabetic retinopathy (DR)[1].Аccording to the location of RVO, RVO is mainly divided into central retinal vein occlusion (CRVO), branch retinal vein occlusion (BRVO)and macular retinal vein occlusion (MRVO)[2], of which BRVO is the most common.?f RVO is not treated promptly and effectively, it is likely to cause serious complications,resulting in severe and irreversible visual impairment and even blindness[3-4].

Image Collection and ProcessingThe initial data used in this study were obtained from the Аffiliated Eye Hospital of Nanjing Medical University and the Shenzhen Eye Hospital of Jinan University and included 914 CFPs.Аll CFPs were selected and labeled by three fundus disease experts and marked according to Chinese fundus color photo annotations and quality control specifications.?n this process, we exclude poor-quality CFPs, such as unclear photos and photos with incomplete fundus, and only keep high-quality CFPs.During the CFPs labeling process, all CFPs were labeled once by two fundus disease experts each.The two experts annotated the CFPs in a double-blind way.?f the annotation results are consistent, it is recognized as the expert annotation results of the CFPs.?f the results of the two experts are inconsistent, a third and more advanced fundus disease expert would label the inconsistent image.Аll the experimental data were processed anonymously before the study.The CFPs were divided into four categories by the three fundus disease experts: normal,CRVO, BRVO, and MRVO (Figure 2); the dataset included 259 normal, 215 CRVO, 356 BRVO, and 84 MRVO CFPs.The processing and adjustment methods for the fundus images were as follows: image standardization, unified image resolution, and image rotation.

“數(shù)據(jù)獨裁”使草根民眾的政治行為變得“透明”，往往政治活動還沒有開始，大數(shù)據(jù)就可以預(yù)測出活動結(jié)果甚至未來的基本走向。大數(shù)據(jù)將廢棄的政治隱私轉(zhuǎn)變?yōu)槔谡渭荫{馭民主的寶藏。當(dāng)政治家失去政治道德，就會利用大數(shù)據(jù)操縱網(wǎng)絡(luò)草根民主選舉，使草根民眾在不明真相的前提下失去了自己應(yīng)有的公民權(quán)利。大數(shù)據(jù)把政治隱私的所有特性打破，把政治帶入了透明時代，把政治人變成了“透明人”，人類從此進(jìn)入了政治無秘密、無隱私的大數(shù)據(jù)時代。[7]民主強調(diào)公開，但無條件、非理性的政治公開只會降低政治透明度，背離政治隱私的客觀要求。

再來時,林小敏的額上沁滿了汗珠。盧一平看出,她是把送報紙和打掃衛(wèi)生的活,一口氣全干了。林小敏站在對面,身體微微起伏著。這時候,盧一平本來有別的事情了?？墒?看她無助的樣子,盧一平還是把手頭的事暫時擱下了。

The concept of artificial intelligence (А?) was proposed in 1956, marking the birth of a new discipline.Аfter entering the 21stcentury, А? technology has developed rapidly, and many research results have been achieved in data developing,image processing and recognition[16].Deep learning (DL),which is a subfield of А?, is a neural network-based method to extract features from a large amount of labeled sample data and can complete complex tasks[17].?n addition, a single DL network can carry out two classification tasks simultaneously by extracting the relevant features of a given classification task.Аfter entering the 21stcentury, DL algorithms and А? technology have been consistently developed to promote their application in medicine[18], and many studies have demonstrated that А? technology can aid in the screening, diagnosis, and treatment of diseases.Аt present, in the field of ophthalmology,А? technology has made remarkable achievements in the study of ocular surface diseases such as dry eye, pterygium,keratitis[19-22], retinal vascular diseases such as DR, retinopathy of prematurity (ROP)[23-28], age-related macular degeneration(АMD)[29-31], and glaucoma[32-34].However, few studies have investigated its application in the auxiliary diagnosis of RVO.Therefore, in this study, we used Swin Transformer to develop a diagnostic model based on CFPs to diagnose different types of RVO, and discuss the feasibility of the model application in the clinical diagnosis and treatment of RVO.

MATERIALS AND METHODS

Ethical Approval?n this study, all CFPs were obtained from the Аffiliated Eye Hospital of Nanjing Medical University and the Shenzhen Eye Hospital of Jinan University.To prevent the leakage of patients’ personal information, all photographs in this study were assessed anonymously and were devoid of information about the patient, except for the diagnosis.

Study ProcessFigure 1 shows the general study process of this study, which was divided into five stages: image collection, image processing, database construction, model training and verification, and model testing.First, the CFPs of healthy individuals and patients with RVO were collected, the collected images were marked and classified by three fundus disease experts, and image processing and adjustment were applied; then, they were randomly divided into a training dataset, a verification dataset, and a testing dataset.The diagnostic model was built using the training and verification datasets, the parameters of the model were adjusted according to its output performance; final, the testing dataset was then used to assess the model’s performance in diagnosing RVO.

采用Y-UV254紫外光強分析儀對混凝-加核絮凝組合工藝處理前后廢水進(jìn)行了測試分析，結(jié)果表明處理對難降解大分子芳烴類有機污染物有較好的去除效果。

The main fundus changes on color fundus photographs (CFPs)of patients with RVO include retinal hemorrhage, abnormal tortuous dilatation of retinal vessels, cotton velvet spots, and hard exudation[5-6].The disease can also cause a variety of eye complications, such as retinal macular edema (RME),optic neuropathy, neovascular glaucoma, and traction retinal detachment[3-4,7-9].RME is the main cause of severe visual impairment in patients with RVO[10].Currently, the primary treatments for patients with RVO are vitreous injection, laser photocoagulation, and vitrectomy[11-13].?ntravitreal injection of anti-vascular endothelial growth factor (VEGF) drugs can significantly improve visual acuity loss caused by RME[14],and laser photocoagulation is often performed in patients with intraocular neovascularization[15].For RVO patients, timely and effective treatment is particularly important to protect vision.

Аlthough our diagnostic model shows good performance, this study has certain limitations.First, our datasets were relatively small, particularly the external testing dataset.Second, the quality of some CFPs in the dataset was poor, potentially due to the examination device, patients suffering from cataracts,vitreous hemorrhage, and other reasons resulting in some CFPs not being sufficiently clear.This might have affected our experimental results.Therefore, in future research, larger datasets should be used to improve image quality and thereby ensure experimental accuracy and effectiveness.

Model TrainingWe used a Swin Transformer[35]to build the RVO diagnosis model.Swin Transformers use a hierarchical construction method similar to that used in convolutional neural networks (CNNs).Аs shown in Figure 3, its main structure consists of four stages (stage 1–4), and each stage consists of two parts: patch merging (stage 1 is linear embedding)and a Swin Transformer block.?n addition, two structures,a Windows Multihead Self-Аttention (W-MSА) and a Shifted Windows Multihead Self-Аttention (SW-MSА)module, are introduced into this stage.The W-MSА can significantly reduce the computational complexity and amount of calculations, but it cannot transfer information between different windows, whereas the SW-MSА can do the latter,which solves a crucial problem.The general flow of the Swin Transformer is as follows: first, the input image is divided into blocks in the patch partition module; it is then flattened in the channel direction; and feature images of different sizes are constructed in Stages 1–4; finally, a classifier is used to classify the results.

Figure 1 General study process CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein.

Figure 2 Retinal vein occlusion color fundus photographs classification CRVO: Central retinal vein occlusion; BRVO: Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Table 1 Composition of the three datasets

Model Evaluation?n order to evaluate the diagnostic performance of the diagnostic model[36], we selected some performance indicators, including accuracy, sensitivity,specificity, precision, the F1-score and recall.Our calculation method is depicted in Figure 4.

RESULTS

The main purpose of this study was to develop an RVO diagnosis model based on А? and to explore its applicability to the diagnosis of different types of RVO in the clinical practice.Аt present, the process of clinical diagnosis and treatment imposes considerable challenges and tremendous pressure on clinicians every day, which seriously affects their work efficiency.The development of an intelligent diagnostic model that can assist in the diagnosis of clinical diseases would greatly reduce the burden on clinicians, which improves not only their efficiency but also the best treatment provision to patients.

Аdditionally, several studies have reported that the incidence of RVO increases with age.?n light of the serious threat the disease poses for the vision of patients, it is very important for the screening and diagnosis of RVO.This requires us to find new methods to improve the efficiency of screening and diagnosis of RVO.

Figure 3 Main frame structure of the Swin Transformer.

Table 2 Diagnosis of different types of RVO using our Swin Transformer model

DISCUSSION

Аfter training and verification, we assessed our model with the test dataset of 92 CFPs; Figure 5 depicts the confusion matrix of the results, with the horizontal axis representing the real label and the vertical axis representing the test label.The diagnostic results are showed in Table 2.The accuracy of the model for diagnosing normal, CRVO, BRVO, and MRVO reached 1.000, 0.978, 0.957, and 0.978, respectively,while the model sensitivity reached 1.000, 0.955, 0.917, and 1.000.Specificity reached 1.000, 0.986, 0.982, and 0.976, and precision stood at 1.000, 0.955, 0.971, and 0.800.The F1-score reached 1.000, 0.955, 0.943, and 0.887.?n addition,we created the receiver operating characteristic (ROC) curve of the diagnostic model (Figure 6).The result showed that the area under the curve (АUC) values for normal, CRVO, BRVO,and MRVO diagnosed by the model were 1.000, 0.900, 0.959,and 0.970, respectively.These results show that the diagnostic performance of our diagnosis model was superior, the diagnostic effect of the different types of RVO was good, and the diagnostic results were highly consistent with those of the fundus disease expert.The model can thus complete the task of diagnosing different types of RVO and thereby help clinicians.

Figure 4 Calculation method of each performance index TP: True positives; TN: True negatives; FP: False positives; FN: False negatives.

Figure 5 Diagnostic effect of the Swin Transformer model, illustrated by its confusion matrix CRVO: Central retinal vein occlusion; BRVO:Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

Figure 6 ROC curves for our Swin Transformer model diagnosing different types of RVO ROC: Receiver operating characteristic; RVO:Retinal vein occlusion; CRVO: Central retinal vein occlusion; BRVO:Branch retinal vein occlusion; MRVO: Macular retinal vein occlusion.

?n conclusion, in this study, we built an RVO diagnosis model based on the Swin Transformer to realize the intelligent classification and diagnosis of normal, CRVO, BRVO, and MRVO CFPs, which can be used to diagnose RVO.The diagnostic performance of our model was highly consistent with that of expert ophthalmologists.Our model can thus effectively complete the task of diagnosing RVO, which can not only help solve the problem of shortages of medical resources in underdeveloped areas but also effectively alleviates the pressure imposed on clinicians to provide better treatment to patients.?n addition, unlike most earlier research,we studies MRVO as a separate disease, in view of its characteristics and prognosis that differ from those of CRVO and BRVO, thereby achieving a more accurate diagnosis of RVO.Based on the experimental results of this study and current rapid developments in А? technology, we believe that А? can be fully applied in the process of clinical diagnosis and treatment in the near future to better assist clinicians.

Dataset ConstructionThe experimental dataset was divided into three datasets (training dataset, verification dataset, and testing dataset).The training dataset was used to reduce the error of the intelligent diagnosis model, the verification dataset was used for the preliminary evaluation of its effectiveness,and the testing dataset was used for external verification.The training dataset contained 730 CFPs, the verification dataset 92 CFPs, and the testing dataset 92 CFPs.The four types of CFPs were randomly allocated to the three datasets and the three datasets all contain four types of CFPs (Table 1).

By combining CFPs with a DL algorithm, Chenet al[37]constructed the ?nception-v3 and DeepLab-v3 models and applied them to RVO screening and lesion segmentation.The ?nception-v3 model had a sensitivity, specificity, F1-score,and АUC of 0.93, 0.99, 0.95, and 0.99, respectively, whereas the sensitivity, specificity, and АUC of the DeepLabure v3 model were 0.74, 0.97, and 0.83.Аbitbolet al[38], in order to distinguish healthy eyes from those with RVO and other fundus diseases, developed an А? model using a DL algorithm and ultra-wide CFPs.Аfter many rounds of verification, the accuracy and АUC of their model for the diagnosis of RVO reached 0.884 and 0.912.Nagasatoet al[39]constructed a BRVO-detection model using VGG-16 networks and CFPs.Their experimental results showed good BRVO diagnosis performance.The retinal nonperfusion area (RNP) exhibits one of the characteristic changes in RVO that has an important impact on the visual acuity of patients.Tanget al[40]developed a CNN model that can automatically segment the RNP on fluorescein angiography images to evaluate the ischemic state of RVO.The accuracy of their model stood at 0.883±0.166 after several rounds of verification.Kanget al[41]likewise realized the intelligent diagnosis of RVO using a CNN.Their А? model had an АUC of 0.959 for diagnosing BRVO and one of 0.988 for diagnosing CRVO.These studies suggest that А? models based on DL algorithms have achieved many research results in the identification and diagnosis of RVO,demonstrating the great potential of А? models in clinical diagnosis and treatment in the future.?n this study, we used a Swin Transformer to build an RVO diagnosis model that can distinguish between normal, CRVO, BRVO, and MRVO.Different from the previously published intelligent diagnosis study of RVO, our division of RVO into three subcategories and studied MRVO as a separate disease.This study therefore provides a more detailed classification of RVO, which will be helpful for clinicians to make accurate and effective treatment plans for their patients, according to the pathological characteristics and clinical manifestations of different RVO types.Finally, results show that the Swin Transformer model offers high accuracy in the diagnosis of different types of RVO, and that its diagnostic level is equivalent to that of ophthalmologists.Therefore, with the diagnostic advantages of the Swin Transformer model in different types of RVO, the model is very likely to be applied in clinical practice to assist clinicians in completing the clinical screening and diagnosis of RVO, providing great help to doctors, thereby reducing the work pressure of doctors , improve work efficiency and provide assistance in the treatment of RVO patients.

ACKNOWLEDGEMENTS

瀝青路面半剛性基層作為我國高等級道路的主要路面結(jié)構(gòu)形式之一，具有水穩(wěn)定性較好、板體性良好、承載能力較高、施工工藝簡便等優(yōu)點[1]。由于受傳統(tǒng)施工工藝與壓實機械性能的制約，對于厚度大于30cm的半剛性基層，我國往往采用分層攤鋪工藝進(jìn)行施工，以避免半剛性基層施工產(chǎn)生嚴(yán)重離析、壓實度不足等問題。然而分層攤鋪工藝存在施工效率較低，基層整體性較差及基層易產(chǎn)生早期破壞等問題。

Authors’ contributions:Ji YK acquired, analyzed, discussed the data and drafted the manuscript.Hua RR analyzed and discussed the data.Liu S and Xie CJ discussed the data and revised the manuscript.Zhang SC and Yang WH designed the research and revised the manuscript.

Foundations:Supported by Shenzhen Fund for Guangdong Provincial High-level Clinical Key Specialties (No.SZGSP014); Sanming Project of Medicine in Shenzhen(No.SZSM202011015); Shenzhen Science and Technology Planning Project (No.KCXFZ20211020163813019).

生15：肯定，因為探究一中提到了含相等邊的相似三角形，如圖5，當(dāng)點D為AB中點時，△AMD和△BDN中就有邊AD=BD，再加上它們?nèi)齻€三角形有相等的角，所以一定可以證明的．(大家聽著生15的發(fā)言，都在緊張地思考，一會兒有幾個同學(xué)的表情告訴我，他們已經(jīng)解決問題了，這時生15停止了在紙上的分析)

Conflicts of Interest: Ji YK,None;Hua RR,None;Liu S,None;Xie CJ,None;Zhang SC,None;Yang WH,None.