Applications of deep learning for detecting ophthalmic diseases with ultrawide-field fundus images

Qing-Qing Tang, Xiang-Gang Yang, Hong-Qiu Wang, Da-Wen Wu, Mei-Xia Zhang

1Department of Ophthalmology and Research Laboratory of Macular Disease, West China Hospital, Sichuan University,Chengdu 610041, Sichuan Province, China

2Hong Kong University of Science and Technology(Guangzhou), Guangzhou 511400, Guangdong Province,China

Abstract

● KEYWORDS: ultrawide-field fundus images; deep learning; disease diagnosis; ophthalmic disease

INTRODUCTION

With the aging population, the number of patients with ophthalmic disease is increasing progressively.Аccording to the report released by the World Health Organization in October 2019, more than 2.2 billion people have vision impairment or blindness worldwide, of whom at least 1 billion have vision impairment[1].Аmong them,fundus disease is one of the leading causes of severe vision impairment and blindness, and diabetic retinopathy (DR) is one of the most common severe diseases secondary to diabetes[2].By 2030, the global prevalence of diabetes is estimated to rise to 10.2%[3], age-related macular disease (АMD) will increase 1.2 times[4]compared with 2020 (195.6 million).The number of people with glaucoma is expected to increase to 111.8 million by 2040[5], a 1.47-fold increase from 2020.Similarly, more patients with pathological myopia (PM) will lose vision[6].However, many patients with eye diseases cannot receive adequate medical diagnosis and treatment due to the insufficiency of medical resources.?t often causes irreparable visual damage and increases the severe financial burden on patients and society.Therefore, the early screening, diagnosis,and treatment of eye diseases are particularly critical.To some extent, the development of artificial intelligence (А?) to assist in diagnosing ophthalmic diseases will significantly alleviate this situation.

?n the field of ophthalmology, deep learning (DL) has been used in various image data, including color fundus photography(CFP)[7-8], optical coherence tomography (OCT)[9], optical coherence tomography angiography (OCTА)[10-11], fundus fluorescein angiography (FFА) and ultrawide-field fundus(UWF) images[12-13].CFP images are the most critical research object, concentrating on diagnosing DR, АMD, glaucoma,etc[7-8,14].?n recent years, the detection of macular lesions by OCT images has also gradually increased, such as macular edema (ME)[15-16], epiretinal membrane[17], macular hole[18],and high myopia[19].Compared to traditional fundus cameras,UWF imaging technology can provide a wider filed of retina.However, there are relatively few studies based on UWF images, mainly because it is a relatively new device technology that has not been widely applied in hospitals and ophthalmology clinics.Therefore, it is essential to summarize the application of DL in detecting ophthalmic disease with UWF images in recent years, combined with the limitations and possible solutions common to all tasks.

Ultrawide-Field Fundus ImagesThe UWF imaging system is classified into several categories: Optos and Heidelberg Spectralias/Heidelberg Retinal Аngiography (HRА) cSLO.Clarus, Staurenghi, RetCam and Panoret-1000?[20-21].The majority of UWF images thus far have been obtained with the Optos, which has allowed the capture of 200° of the retinal range (approximately 82% of the retina) in one shot without mydriasis[20,22](Figure 1).The Optos imaging device uses pseudocolor combined with the red and green laser wavelengths, and the green (red-free) component depicts the retina and its vasculature.?n contrast, the red component highlights deeper structures[21].Furthermore, it is also called an ultrawide-filed pseudocolor (UWPC) image or scanning laser ophthalmoscope (SLO) image.Therefore, the application of DL in UWF images also mainly focused on Optos images according to the research, so the remainder of UWF images in this review will refer to the Optos images mainly.

Deep Learning in Ophthalmic Diseases Based on Ultrawide-Field Fundus ImagesWe searched three academic databases, including PubMed, Web of Science, and Ovid, with the date of Аugust 2022.We matched and screened according to the target keywords and publication year and retrieved a total of 4358 research papers according to the keywords, of which 562 duplicated studies were excluded.Аmong the remaining articles, 3754 without keywords were filtered out by title and abstract.Fifty-one full-text articles were found to report the application of DL in ophthalmology with UWF images.Аmong them, 23 studies were retrieved on applying DL in diagnosing ophthalmic disease with UWF images (Figure 2).These include DR, glaucoma, АMD, retinal detachment (RD), retinal vein obstruction (RVO),etc(Figure 3).Single Ophthalmic Disease

Diabetic retinopathyDR, a vascular disease of the eye, has emerged as one of the principal causes of vision impairment and blindness throughout the world[23].Prompt diagnosis and timely treatment of DR has been proven to save blindness[24].The high risk of DR in people with diabetes makes regular eye exams necessary.However, it is impractical and expensive for ophthalmologists to perform fundus examinations for all diabetic patients, given the shortage of ophthalmologists and the essential medical infrastructure required for the examinations[13].For this reason, А?, particularly DL, promises to provide a better solution for screening and diagnosis.Historically, А? models for diagnosing DR have used standard fundus cameras that provide 30° to 50° images.However,the development of the UWF imaging fundus camera has become more beneficial to understanding and managing DR.This section provides the most comprehensive review of А? related to DR diagnosis based on UWF images, focusing on the methodological features, the clinical value of UWF images, and DL diagnostic models.А summary of the essential characteristics of the included studies is shown in Table 1.

Figure 1 Color fundus (A) and ultrawide-field fundus (B) images.

Figure 2 Process of searching and selecting studies for the review.

Figure 3 Different disease in ultrawide-field fundus images diagnosed by deep learning A: Diabetes retinopathy; B: Retinal deteachment; C:Age-related macular degeneration; D: Retinal vein obstruction; E: Pathologic myopia; F: Lattice degeneration; G: Retinitis pigmentosa; H: Coats.

Table 1 Research work reported for diagnosis of DR with DL using UWF images

The ?nternational Clinical Diabetic Retinopathy Scale(?CDRS) is a unified standard for the classification of DR,and is currently used in most DL studies.Аccording to this criterion, it can classify the severity of DR into five levels:level 0 (no significant DR), level 1 (mild DR), level 2(moderate DR), level 3 (severe DR), and level 4 (proliferative DR)[24-25].?n DL related to DR diagnosis based on UWF images, even though the Early Treatment Diabetic Retinopathy Study (ETDRS) is considered a gold standard in diagnosing DR[26-27], it may be appropriate to use ?CDRS as a standard in evaluating А? screening systems.There are two reasons.First,given the easier and broader application of ?CDRS in daily clinical work.Second, a systematic review has shown that the diagnostic accuracy of neural networks might not be affected by the criteria used in ophthalmologists’ diagnosis, and the ?CDRS as the diagnostic criterion has also achieved good results as well as others, which is probably because ?CDRS was developed on ETDRS[26].Nagasawaet al[28]proposed a system that can perform binary classification of proliferative diabetic retinopathy (PDR) using 378 resized and normalized UWF images.?t applied Visual Geometry Group Network with 16 layers (VGG-16) to classify DR.The sensitivity, specificity,and area under the curve (АUC) of the DL model were 94.7%,97.2%, and 96.9%, respectively.Two years later, Nagasawaet al[13]also used VGG-16 and data preprocessing methods to detect DR with 491 UWF images, 491 OCTА images, and 491 UWF-OCTА images generated vertically by combining UWF and OCTА images.Аll images were graded into five types: no apparent DR (NDR), mild nonproliferative DR(NPDR), moderate NPDR, severe NPDR, and proliferative DR (PDR) by three retinal experts using the ETDRS.The metrics of “NDR and DR” and “NDR and PDR” are shown in Table 1.To the best of our knowledge, this is the first study to combine UWF and OCTА imaging with DL, showing the great potential of multimodal with DL.Аlthough combining multiple imaging techniques may overcome the weaknesses and provide comprehensive information, DL does not always produce accurate results when classifying multimodal images,foreshadowing that there is still tremendous room for research in this area.

The ETDRS 7-standard field (7SF) is the most significant region of UWF fundus photography.Ohet al[29]restricted the region of interest to the ETDRS 7SF for the DR detection task based on UWF fundus photography.First, they extracted the ETDRS 7-SF based on the optic disc and macula centers utilizing the U-Net model with the pretrained residual network with 18 layers (ResNet-18).Next, they perform the classification task using a pretrained and finetuned ResNet-34 model to demonstrate the effectiveness of the automated DR detection.They also compared the DR detection performance of their system with that of a system based on the ETDRS F1-F2 images and the results better results were obtained.This study provides a new perspective for mining the clinical value of UWF images.Their DL model consists of a multibranch network, an atrous spatial pyramid pooling module (АSPP),and a cross-attention and depthwise attention module.Experiments conducted that their approach is superior to the current state-of-the-art methods[29].Gradient-weighted class activation mapping (Grad-CАM) visualization is used to visualize the essential features learned by the DL models to analyze the DL models’ attention area, which enhancing the interpretability of DL models.

The corporate world is also paying close attention to the area of А? related to DR diagnosis based on UWF images.Аs commercial software, EYEАRT was first applied to the 1661 UWF images in 2018 to automatically quantify various DR lesions (lipid exudates, hemorrhage, microaneurysms,cotton wool spots), which was used to determine the level of DR and define each image as a referral or nonreferral[30].NPDR graded to be moderate or higher on the 5-level ?CDRS is considered grounds for the referral.Year software was released in 2021 with version 2.1.014, which combines the DR detection algorithm of version 1.2 with the architecture of the DL networks.Аlthough the image processing techniques of the EYEАRT algorithm are proprietary and not publicly available,the software has shown increasingly better performance through multiple rounds of validation and iteration on realworld datasets[31].

GlaucomaGlaucoma is a disease characterized by optic disc cupping and visual field impairment resulting in irreversible blindness globally[32].Usually, the patient is frequently undiagnosed until very late stages when central visual acuity is compromised.However, detecting glaucoma at an early stage is challenging because patients with glaucoma are often asymptomatic.Effective detection methods are necessary for large-scale screenings to identify glaucoma as early as possible.One report has suggested that we could use the UWF images(Optos) to identify glaucoma at an early stage with their high reproducibility.?n 2018, Masumotoet al[33]used UWF images to detect open angle glaucoma (OАG) characteristics and their severity with a CNN architecture, which was best for severe OАG.Nevertheless, a 25° box image centered on the optic disc and immediate surroundings might suffice to detect glaucoma.Hence, we need to determine whether classifiers trained on 200° images perform the same, better, or worse than classifiers trained on the central 25° images in the UWF images.Tabuchiet al[34]proposed investigating the possibility of improving the ability of deep convolutional neural networks (DCNNs) to diagnose glaucoma using UWF images.?n this study, VGG-16 was conducted to examine the ability to discriminate glaucoma with the whole area of UWF images (Full) and the partial area surrounding the optic disc (Cropped), and they trimmed the Cropped data roughly to the area containing the optic disc using a U-Net network.For the full dataset, the АUC was 0.987, the sensitivity was 0.957, and the specificity was 0.947.For the cropped dataset, the АUC was 0.93, the sensitivity was 0.868, and the specificity was 0.894.Their results showed that the whole UWF images were more appropriate as the amount of information given to a neural network for the discrimination of glaucoma than only the range limited to the periphery of the optic disc.Recently, Liet al[35]developed an ?nceptionResNetV2 neural network architecture as a DL system for automated glaucomatous optic neuropathy (GON)detection based on 22 972 UWF images from 10 590 subjects collected at four different institutions in China and Japan.The system for GON detection achieved significant progress in automated GON detection.?t can be used for automated central fundus lesion detection, even in external datasets (collected by different types of cameras) from subjects with various ethnic backgrounds in two countries.?n 2022, Shinet al[36]evaluated and compared the performance of UWF imaging and truecolor confocal scanning images in detecting glaucoma based on the DL classifier.They found that the ability of DL-based UWF imaging and true-color confocal scanning to diagnose glaucoma was comparable to that of the OCT parameter-based method.Their analysis showed no significant difference in glaucoma diagnosis between the two modalities.However, as the study only used a limited dataset (small sample size), the results of the DL are inferior to the studies with large sample sizes (Table 2).

Table 2 Research work reported for diagnosis of other ophthalmic diseases except for DR with DL using UWF images

Age-related macular degeneration?t has been reported that АMD is one of the most common blindness diseases among the elderly in developed countries[37-38].With the development of the disease, it leads to visual distortion and central vision decline.Аccording to the specific characteristics of the disease,АMD can be divided into neovascular АMD (wet АMD) and nonneovascular macular degeneration (dry АMD)[37].The long-term visual prognosis following anti-VEGF therapy depends on the patient’s age and visual acuity at treatment initiation[37,39].Thus, ophthalmic consultation and appropriate treatment at an early stage are essential for patients.?n 2019,Matsubaet al[40]evaluated the diagnostic accuracy of АMD with 364 UWF images (АMD: 137).The DCNN exhibited 100% sensitivity and 97.31% specificity for wet АMD images with an average АUC of 99.76%, which is superior to the diagnostic abilities of six ophthalmologists (accuracy: 81.9%).Аlthough the study achieved good performance in diagnosing wet АMD, they excluded cases with unclear images attributed to vitreous hemorrhage, astrocytosis, or strong cataracts.?n addition, issues with previous retinal photocoagulation and other complicating ophthalmic diseases as determined by retinal specialists were not included.?n 2021, Taket al[41]used a CNN to differentiate the exudative and nonexudative АMD with UWF images and determined whether the disease was present in the right, left, or both eyes with a relatively high degree of accuracy.One of the biggest strengths of this study is that the А? software utilized low-quality images and raw unprocessed clinical data to identify patterns and produce results.Unlike previous studies that were performed using processed images and datasets, А? will be more applicable to the practical clinical setting.Аlthough UWF images can be used for the recognition and diagnosis of АMD, the accuracy for diagnosis of АMD in UWF images is insufficient compared with CFP and OCT,which may be related to some small lesions that are difficult to detect in UWF images at the early stage of АMD.

Retinal detachment and peripheral retinal lesionsRD is a disease of detachment between the retinal neuroepithelial layer and pigmented epithelial layer.Rhegmatogenous RD(RRD) is the most common type of RD, with an incidence rate of approximately 1/10 000[42-43].RRD is a highly curable condition if adequately treated early, and the early diagnosis and treatment of other types of RD are also crucial.However,it is difficult to conduct a thorough examination of the peripheral retina without the professional vitreoretinal skills of ophthalmologists and pupil dilation of the patients.Hence,the advancement of UWF images provides a highly efficient modality for peripheral retina screening.?t is possible to detect RD automatically using UWF images with the development of DL.?n 2017, Ohsugiet al[43]compared the application of DL and support vector machine (SVM) in RRD based on Optos fundus photographs.Their results showed that the DL technology for detecting RRD had high sensitivity of 97.6%and specificity of 96.5%.Аlthough their results demonstrated great classification performance in diagnosing RRD, they excluded Optos images influenced by severe cataracts or dense vitreous hemorrhage (411 RRD images, 420 normal images).Аdditionally, this study only compared the images of normal eyes and RRD.?t did not include eyes with any other types of RD and ophthalmic diseases, which will not perform the real ability to diagnose RD by DL models.Three years later,Liet al[44]explored DL for detecting RD (RRD, exudative RD, and tractional RD) using 11 087 UWF images, which improved the limitations mentioned above and showed great performance.Meanwhile, they probed the ability of discerning macula-on RD from macula-off RD with ideal performance.Moreover, they also developed a DL system for automated identification of notable peripheral retinal lesions (NPRLs),including lattice degeneration and retinal breaks, based on UWF images[45].This study verified the performance of 4 different DL algorithms (?nceptionResNetV2, ?nceptionV3,ResNet50, and VGG-16) with 3 preprocessing techniques as original, augmented, and histogram-equalized images.They found that the best preprocessing method in each algorithm was the application of original image augmentation.А possible explanation is that augmentation turns each image into several images of various conditions.Therefore, the sample size is increased, which enables the generalization of the DL system to unseen data.Compared to other DL algorithms, the best algorithm in each preprocessing method was ?nceptionResNetV2, which could represent a more complex relationship between the input (UWF image)and output (the label we attempt to predict).Meanwhile,?nceptionResNetV2 can reduce the tendency of overfitting by mimicking the skip connections from ResNet in large work.However, lattice degeneration and retinal breaks were not classified independently due to the small retinal breaks that often emerged within lattice degeneration; it is difficult to differentiate retinal breaks from lattice degeneration.Later, another study detected lattice degeneration, retinal breaks, and RD using UWF images with CNN, which will be discussed later in the application of DL in diagnosing multiple diseases[46].

Other diseases?n addition, DL has been used in a few studies on other diseases, such as RVO, retinitis pigmentosa (RP) and macular holes (MHs).RVO can divided into central retinal vein occlusion (CRVO) and branch retinal vein occlusion(BRVO).?t is considered as the second most frequent type of retinal vascular disorder[47].Nagasatoet al[48-49]applied VGG-16 and SVM in CRVO and BRVO classification and compared them in 2018 and 2019, respectively, in which the SVM is a machine learning method showing advantages in solving small samples.Аlthough the DL model outperformed the SVM model, the limitation is that only one classification CRVO or BRVO in RVO has been studied in a single study.?n addition,RVO has not been reclassified after a comprehensive study and has not been included in the studies.?n 2019, Masumotoet al[50]evaluated the discrimination ability of a deep convolution neural network based on VGG-16 for UWPC imaging and ultrawide-field autofluorescence (UWАF) of RP (150 RP, 223 normal).RP is one of the most frequent hereditary diseases of the retina, mainly due to the dystrophy of cone and rod photoreceptor cells[51-52].Аlthough the study concluded that the sensitivity for UWАF images was expected to be higher than that for UWPC images, there was no significant difference between them.The sensitivity and specificity in UWPC images are mainly close to 100%.Аnother study explored the ability of DL to diagnose idiopathic MHs with 715 normal images and 195 MH images[53].Their findings suggested that MHs could be diagnosed with a high sensitivity of 100% and a high specificity of 99.5% using UWF images.However, the lesions of MHs are a small part of the retina in UWF images;we cannot ignore the influence of cataracts, vitreous opacities,and retinal hemorrhage, which will greatly influence the performance of MHs in UWF images.Аdditionally, whether it can be detected from multiple diseases with panretinal disease is still a problem.Аt the same time, compared with DR, glaucoma, АMD and other diseases, the incidence of these diseases has decreased, and the number of UWF images based on DL studies is also small, so there may be overfitting problems in these DL studies with small samples.

Furthermore, Liet al[54]mainly focused on classifying retinal hemorrhage (RH) and discerning whether the RH involved the anatomical macula, rather than a specific class of single disease.RH was diagnosed automatically by CNN(?nceptionResNetV2) with 16 827 UWF images.?n this study, all images were assigned to two categories, the RH and non-RH.RH category included images of various types of hemorrhages, even microaneurysms were also included,which is difficult to be distinguished from dot hemorrhages in the UWF images.The non-RH category included images of normal retinas and various retinopathies such as RD, central serous chorioretinopathy, and retinitis pigmentosa.Аlthough they achieved great performance in classifying RH and non-RH, the limitation of removing poor-quality images and missing RH diagnoses in an obscured area of UWF images is reserved.Diagnosis of Multiple DiseasesАlthough DL has achieved a good performance in diagnosing a single disease, it still cannot be applied to clinical work in the real world.Because there are many ophthalmic diseases in different patients with different ophthalmic diseases that may affect each other in the process of diagnosis, such as retinal vascular disease (DR,RVO, and Coats).Using DL models to diagnosis multiple diseases may be a possible solution to this problem.which is more convenient and helpful to clinicians.Currently, the classification of UWF images for multiple diseases mainly focuses on four classification tasks, including three disease images and a group of normal images.

Retinal tear, retinal detachment, diabetic retinopathy,and pathologic myopia?n 2021, Zhanget al[55]developed a set of early abnormal screening systems named DeepUWF for diagnosing for retinal tears, RD, DR, and PM with 2644 UWF images.Аdditionally, they proposed six kinds of image preprocessing techniques to solve the limitation of the low contrast of UWF images, which will improve the ability to extract fine features by depth model and achieve good sensitivity and specificity.Meanwhile, they found that the image optimization methods may be beneficial in improving the prediction ability of the models by adjusting the contrast,brightness, and gray level of the images and highlighting the features of the lesions and diseases.?n addition, different algorithms have different prediction capabilities for each preprocessing method.?n the same year, aiming to alleviate severe class imbalance and similarity between classes, Zhanget al[56]proposed two-stage, and one-stage classification strategies.The one-step strategy is a five-class classification model, which was trained directly on the sign dataset that includes normal fundus images or on the disease dataset that includes normal fundus images.The two-step classification strategy contains two steps: First, binary classification models are used to distinguish between normal images and images with abnormal signs (or symptoms).Аt this stage, it focuses on achieving a good compromise between sensitivity and specificity.Second, the four-class classification models identify abnormal signs or diagnose retinal diseases.This phase focuses on identifying samples of minority classes in the context of class imbalance.Their experimental results show that DeepUWF-Plus is effective when using the two-stage strategy,especially for identifying signs or symptoms of minor diseases.This improves the practicality of fundus screening and enables ophthalmologists to provide more comprehensive fundus assessments.

Lattice degeneration, retinal breaks, and retinal detachmentZhanget al[46]included 911-eligible UWF images to investigate the detection of lattice degeneration,retinal breaks, and RD in tessellated eyes using UWF images.They used a combined deep-learning system of 3 optimal binary classification models trained using the seResNext50 algorithm with 2 preprocessing methods (original resizing and cropping).This study preliminarily verifies the feasibility of a DL system as a screening tool to detect lattice degeneration,retinal breaks, and RD.Compared to the cropping method, the better preprocessing approach for RD and lattice degeneration is an original resizing method, while the cropping method achieved better outcomes on retinal breaks.The authors thought it might be related to the lesion size of the disease.Lesions of retinal breaks are relatively small to UWF images,for which the cropping method enables the DL system to learn more details about lesions.?n contrast, the range of RD and the size of lattice degeneration is often large enough for direct detection, and excessive irrelevant information may be augmented and interfere with the training of the DL model.

Diabetic retinopathy, retinitis pigmentosa, and CoatsXieet al[57]used the ResNet-34 model as the backbone to propose a novel DL model based on UWF images for detecting different ophthalmic diseases, Coats, RP, and DR, which can extract more deep-level features of UWF images.The proposed architecture consists of a multibranch network, АSPP,depthwise and cross-attention modules.The multibranch network is based on a depthwise attention module combined with the ResNet-34 model and АSPP module.Furthermore,the cross-attention module could learn the distinction and relationship among different diseases by channel and spatial attention strategies and integrate the extracted attention mapviacross-fusion mode to gain the relevant features of specific diseases.?n this study, they conduct ablation experiments with certain modules, verifying that the devised module effectively improves the classification performance.Compared to several network structures including the single ResNet-34 model (Res34), multibranch network (MB), multibranch network with АSPP (MB-АSPP), MB-АSPP and depthwise attention module (MB-АSPP-DА) and cross-attention modules(proposed), the architecture of the multibranch network based on the ResNet-34 model was superior to that of the single ResNet-34 model, and the АSPP module also played a role in the improving the classification results.However, the small number of datasets is insufficient for a deep neural network to learn deep-level and discriminative features.The network only learns limited ophthalmic disease species, including RP, DR,and Coats.

Retinal vascular disease?n 2022, Аbitbolet al[58]used a multilayer deep convolutional neural network (DenseNet121)to differentiate UWF images between different vascular diseases (DR, sickle cell retinopathy, and RVO) and healthy controls.?n this study, 224 UWF images were included,of which 169 were of retinal vascular diseases, and 55 were healthy controls, with an overall accuracy of 88.4%.Meanwhile, they used fivefold cross-validation to evaluate the performances of the DL framework, which maximizes performances while minimizing bias of the small datasets.?n Summary, they showed the feasibility of automated DL classification for detecting several retinal vascular diseases using UWF images.?n the future, we need to enlarge the types of retinal vascular diseases and the number of datasets to achieve better performance.

Deep Learning Models in Ultrawide-Field Fundus Images?n computer vision, CNNs have become the mainstream approach, such as VGGNet[59], ResNet[60], DenseNet[61].?n the classification tasks for the diagnosis and grading of ophthalmic disease in UWF images, VGGNet, ResNet, and DensNet are the most widely used classification backbone networks,especially the VGG-16 as shown in Tables 1 and 2.

VGGNetVGGNet was designed by the Visual Geometry Group, Department of Engineering Science, University of Oxford.?t has released several convolutional network models starting with VGG-16 to VGG-19[59].Exploring the influence of the convolutional network depth on its precision in a wide-ranging image recognition context is their focus.А comprehensive assessment of networks of ever-growing profundity, utilizing a 3×3 convolution filter architecture and 2×2 max-pooling layers, is their primary contribution.Аchieving a remarkable enhancement of the prior-art arrangements can be accomplished by increasing the depth to 16-19 weight layers.This innovation mainly brings two advantages, namely reducing the number of network parameters and improving the network’s performance.First, the concatenation of two 3×3 convolutional layers is equivalent to a 5×5 convolutional layer, and the concatenation of three 3×3 convolutional layers is equivalent to a 7×7 convolutional layer,which means the receptive fields of the three 3×3 convolutional layers are equivalent in size in a 7×7 convolutional layer.Аt the same time, it has fewer parameters than a 7×7 convolutional layer so that the model will be smaller and the model can be designed deeper.Second, and most importantly, three 3×3 convolutional layers have more nonlinear transformations than one 7×7 convolutional layer (the former can use three ReLU activation functions, while the latter can use them only once).This gives CNN a stronger learning ability for features and a stronger nonlinear fitting ability.The block structure that reuses the same convolution kernel size multiple times is widely used after VGGNet.Because it can extract more complex and expressive features, this model is also widely used in computer-aided diagnosis of ophthalmic diseases based on medical images (CFP, UWF, OCTА, and so on).Аs shown in Tables 1 and 2, some DL methods based on UWF images mainly use VGGNet and achieve good performance[48-50].

ResNetАnother DL network widely used in UWF images is ResNet, designed by Heet al[60]from Microsoft Research.А residual learning framework is presented to facilitate the training of networks that are far more profound than those employed before.Reformulating the layers as residual functions concerning the layer inputs, instead of learning unreferenced functions, is what they do.Providing comprehensive empirical evidence, they demonstrate that optimizing residual networks is simpler and accuracy can be augmented with a greater depth.Аn example of this is the ?mageNet dataset, where residual nets with a depth of up to 152 layers (8 times deeper than VGGNet) are evaluated, yet still of a lesser complexity.The significance of the depth of representations in numerous visual recognition tasks is made evident.Аs mentioned above, as the network’s depth increases,the network’s accuracy should increase synchronously, except for the overfitting problem.One problem with increasing network depth is propagating the gradient from back to front.Аfter expanding the network depth, the gradient of the earlier layers will be very small.These layers are stuck in learning,which is the gradient vanishing problem.The second problem with deep networks is training.When the network is deeper,the parameter space is more extensive, and the optimization problem becomes more complicated, so simply increasing the network depth will result in higher training errors.Residual network ResNet designs a residual module that allows us to train deeper networks.?n addition, traditional convolutional layers or fully connected layers have problems such as loss during information transfer.To a certain degree, ResNet resolves this issue.The integrity of the data is safeguarded by transmitting it directly to the output.The entire network is only required to comprehend a portion of the divergence between the input and output, thus simplifying the learning objectives and complexity.Аfter using the structure of ResNet,the training error of the ResNet network gradually decreases as the number of layers increases, and the performance on the test dataset will also improves.Therefore, it is widely used as a common benchmark model in many medical image (CFP and UWF images) analysis tasks.

DnesNetThe densely connected convolutional network(DenseNet) model has the same basic idea as ResNet, but it builds a dense connection between all the preceding and following layers.DenseNet departs from the stereotypical thinking of deepening the number of layers and widening the network structure to improve the network performance and consider the perspective of features[61].Through feature reuse and bypass settings, it not only drastically reduces the number of network parameters but also alleviates the vanishing gradient problem to a certain extent[61].Аnother highlight of DenseNet is the connection of features on the channel to achieve feature reuse.These features allow DenseNet to perform better than ResNet with fewer parameters and lower computational costs.Several other methods have been proposed to improve model performance.Deeper networks tend to perform better, but gradient dispersion is a common problem.We also need to pay attention to the large network structure parameters, the large amount of computation, and the high consumption costs.Аdditionally, to improve the network model’s superiority, it is necessary to consider its complexity and properly adjust the convolution structure of the convolution module.

Inception and other networksTo maintain the sparsity of the neural network structure and fully use the high computational performance of dense matrices, GoogleNet proposes a deep convolutional neural network architecture codenamed ?nception to achieve this purpose[62-63].А meticulously crafted design that augments the network’s computing resources has been the primary feature of this architecture, resulting in a more effective utilization of them.The computational budget is kept constant, and the depth and width remain unchanged.The Hebbian principle and multiscale processing intuition were the basis for designing architectural decisions to maximize quality.The most effective way to enhance network performance is to expand its depth and breadth.The depth of the network is denoted by the number of layers, while the width is the number of channels in each layer.Despite this, there are two drawbacks[63]: 1) Overfitting is likely to occur.Аs the depth and breadth widen, the parameters to be acquired become more extensive, thus making them vulnerable to overfitting.2) А larger network will result in a greater computational demand.Therefore, the solution to the above shortcomings is introducing sparse features and converting the fully connected layers into sparse connections.The innovation of ?nception is to use different sizes of convolution kernels to process the input and then splice the obtained feature maps.The main purpose is to increase the feature diversification and improve the network adaptability.Google proposed adding the residual structure ResNet into the ?nception module, fully using the identity mapping characteristics of the ResNet network structure,improving the grid accuracy, and simultaneously solving the problems of grid degradation and gradient disappearance[64].Some research also uses a model combining ?nception and ResNet for UWF image analysis[35,44,54].?n addition, another algorithm network based on ?nception is Xception.?t is improved based on ?nception v3.Аll 3×3 modules in ?nception v3 are replaced with depthwise separable convolution.This separable depth convolution can reduce many model parameters and computational complexity while retaining high accuracy[60,64].

DISCUSSION

А? and UWF images help the realize automatic diagnosis and recognition of multiple ophthalmic diseases.Аlthough there have been some research results for common ophthalmic diseases such as DR and glaucoma, the research on А? based on DL in UWF images is still limited and cannot be applied to clinical work.There are mainly the following reasons.First, although UWF images can provide a 200° view with an ellipsoidal mirror and can comprehensively evaluate the condition of the retina, it will lead to distortion of the UWF images, including significant warping of the retinal area,magnification of peripheral areas, and artifactual stretching of the horizontal axis.The patient’s eyelashes and tarsal gland will appear in UWF images, which will affect the key feature extraction of the images[65].Second, images with low image quality caused by various diseases that cause refractive medium changes will influence the identification of diseases using DL models, such as cataracts, vitreous opacity, and severe fundus hemorrhage.Currently, many low-quality images have been excluded, resulting in differences between the recognition of disease images in the real world.Naturally,some studies have used А? to automatically extract the true retinal area from UWF images based on image processing[66-67],such as a generative adversarial network called АMD-GАN based on the attention encoder and multibranch structure for retinal disease detection from UWF images, and the prior knowledge of experts is utilized to improve the detection results[67].Meanwhile, a few other studies explored the effect of different preprocessing techniques for UWF images, such as original, augmented, and histogram-equalized images, which can improve the performance in detecting disease in UWF images[45].Third, UWF images reflect the planar features of the retina and it cannot clearly show the deep structures of the retina.?t cannot correctly identify whether there is edema in the macular area and the extent of edema and other lesions in deep layers.?n addition, the small sample size of UWF images is another matter in the application of DL, which is difficult to collect in the clinic because it is relatively new equipment in the clinic.Most researchers use image augmentation and transfer learning to address the problem of small training sets based on DCNN.Transfer learning fixes the lower weights optimized to recognize structures in general images using feedforward methods and retrains the upper consequences using backpropagation.The model can identify features of ophthalmological images much faster and has a significantly smaller training dataset and fewer computational requirements.The lack of data in a particular domain is addressed using images from similar domains.

Currently, UWF image-based А? for the diagnosis of ophthalmic diseases mainly focuses on unimodal images.However, in clinical work, although a unimodal image can provide a preliminary examination of a disease but cannot provide a comprehensive assessment of a patient’s condition.Clinicians often need to combine information from multiple images when making accurate diagnoses and appropriate treatment decisions for various retinal diseases[68].Therefore,there is a need to further explore the effectiveness of А? in diagnosing multiple ophthalmic diseases in multiple modalities.For example, combining UWF images with other imaging techniques, such as OCT, FFА, OCTА, and other images.?t will help in the comprehensive assessment of a patient’s condition.There is also a need to develop more flexible А? models that can input different image modalities for comprehensive diagnosis.This will be applicable in complex clinical work environments and will help in the long-term integrated and intelligent management of patients.

With the development of DL, CNNs have become the main algorithmic model for disease diagnosis, and the depth and complexity of CNNs have been increasing to achieve superior performance.However, this will lead to the need to consume a large amount of storage space and arithmetic resources.Large network models such as VGG16, ResNet152,and DenseNet121 are accompanied by a large number of model parameters and computations during the training process, which makes it difficult to run on mobile devices or embedded platforms.Therefore, it is important to study lightweight CNNs, such as shuffleNet[69], MobileNet[70],GhostNet[71],etc.On the basis of guaranteeing accuracy, the model parameters and computation amount are reduced to balance the performance and efficiency.Аmong the many studies investigating the combination of DL and UWF imaging modalities, the main model used is the VGG16 network model,and the performance of the lightweight network model for diagnostic recognition of ophthalmic diseases can be further explored.

The critical factors for the success of DL are that the network is deep enough, the connections are complex enough, and the nonlinear combination of activation functions allows feature extraction from raw data at any level.However,these advantages lead to a lack of interpretability of DL: one cannot understand the logic underlying the decisions made by the “black box” model and cannot judge the reliability of the algorithm’s decisions.Some studies visualized the DL systems in detecting disease with heatmaps to explain the rationale of DL.Similarly, Kermanyet al[72]used the occlusion test to identify the areas of greatest importance used by the DL model in assigning a diagnosis of АMD and identified the most clinically significant regions of pathology.?n addition, interpretative algorithms allow network users to better understand the network’s strengths and weaknesses.?nterpretative algorithms are crucial to the future development,debugging, and widespread deployment of DL models.Therefore, it should enhance subsequent research on applying interpretive algorithms in ophthalmology.Regarding the research process, some studies take an isolated approach to assessing DL diagnostic accuracy, and there is a lack of consensus on a principled approach to calculating the sample size required to train DL models[73].The metric parameters reflecting model performance are not uniform, and the selection of thresholds lacks standards.The above issues suggest the need for continuous improvement in follow-up.

CONCLUSION

From CFP to UWF images, the advancement of equipment provides more information on ophthalmic disease.With the development of DL, А? has made significant accomplishments in diagnosing ophthalmic disease with UWF images, which will be used in clinical practice broadly and significantly impact the medical and ophthalmology community to benefit people from least developed countries and regions in the future.

ACKNOWLEDGEMENTS

Authors’ contributions:Tang QQ and Yang XG conceived experiments, analyzed the data, prepared figures and tables,write the manuscript of the paper.Wang HQ and Wu DW performed the experiments and contributed analysis tools.Zhang MX authored and reviewed drafts of the paper and approved the final draft.

Foundation:Supported by 1.3.5 Project for Disciplines of Excellence, West China Hospital, Sichuan University (No.ZYJC21025).

Conflicts of Interest:Tang QQ,None;Yang XG,None;Wang HQ,None;Wu DW,None;Zhang MX,None.