Multi-View Auxiliary Diagnosis Algorithm for Lung Nodules

Shi Qiu,Bin Li,Tao Zhou,Feng Li and Ting Liang

1Key Laboratory of Spectral Imaging Technology CAS,Xi’an Institute of Optics and Precision Mechanics,Chinese Academy of Sciences,Xi’an,710119,P.R.China

2School of Information Science and Technology,Northwest University,Xi’an,710127,P.R.China

3School of Computer Science and Engineering,North Minzu University,Yinchuan,750021,P.R.China

4Institute of Education,University College London,London,The United Kingdom

5Department of Radiology,The First Affiliated Hospital of Xi’an Jiaotong University,Xi’an,10061,P.R.China

Abstract: Lung is an important organ of human body.More and more people are suffering from lung diseases due to air pollution.These diseases are usually highly infectious.Such as lung tuberculosis, novel coronavirus COVID-19,etc.Lung nodule is a kind of high-density globular lesion in the lung.Physicians need to spend a lot of time and energy to observe the computed tomography image sequences to make a diagnosis,which is inefficient.For this reason, the use of computer-assisted diagnosis of lung nodules has become the current main trend.In the process of computer-aided diagnosis,how to reduce the false positive rate while ensuring a low missed detection rate is a difficulty and focus of current research.To solve this problem, we propose a three-dimensional optimization model to achieve the extraction of suspected regions,improve the traditional deep belief network,and to modify the dispersion matrix between classes.We construct a multi-view model,fuse local three-dimensional information into two-dimensional images, and thereby to reduce the complexity of the algorithm.And alleviate the problem of unbalanced training caused by only a small number of positive samples.Experiments show that the false positive rate of the algorithm proposed in this paper is as low as 12%,which is in line with clinical application standards.

Keywords: Lung nodules; deep belief network; computer-aided diagnosis;multi-view

1 Introduction

Lung nodules are the main lesions of the lungs.If they are not detected and treated in time,the malignant lung nodules will be transformed into lung cancer, which will seriously affect human life and health[1-3].Lee et al.[4]proposed the random forest classification of lung nodules.Wu et al.[5]constructed a hierarchical learning network to extract lung nodules.Diciotti et al.[6] constructed a morphological model to segment lung nodules.Li et al.[7] established Principal Components Analysis(PCA)model to identify lung nodules.Song et al.[8]used the Local Binary Pattern(LBP)features to identify lung sessions.Song et al.[9]established a local optimal classification network to identify lung nodules.Teramoto et al.[10]used the cylindric nodule-enhancement filter to enhance the image information of lung nodules.Tariq et al.[11]introduced neuro fuzzy to identify lung nodules.De Carvalho Filho et al.[12]used quality threshold clustering,genetic algorithm and diversity index to detect solitary lung nodules.Parveen et al.[13] used Support Vector Machine (SVM)kernels to classify lung nodules.Hua et al.[14] applied the theory of deep learning to the classification of lung nodules.Shen et al.[15] established a multi-scale convolutional neural network to classify lung nodules.Sun et al.[16] distinguished lung nodules based on Three-Dimensional (3D)texture features of lung.Javaid et al.[17] distinguished the signs of lung nodules from the gray, geometric,statistical point of view.Qiu et al.[18]used Gestalt to detect nodules.Huang et al.[19]detected the lung nodules with 3D conventional neural networks.Shaukat et al.[20]combined multiple features to reduce the false detection rate of lung nodules.Han et al.[21] established a system of diameter and volume to judge benign and malignant lung nodules.Nishio et al.[22] used gradient tree boosting and Bayesian optimization to assist in the diagnosis of lung nodules.Xie et al.[23]realized automatic classification of lung nodules by fusion of multiple features at the decision-making level.Saien et al.[24]proposed sparse field level sets and boosting algorithms to reduce the false detection rate of lung nodules.Qiu et al.[25]detected lung nodules on Computed Tomography(CT)image.Qiu et al.[26]detection of solitary lung nodules based on brain-computer interface.Rey et al.[27]used CT studies based in soft computing to achieve lung nodule segmentation.Mittapalli et al.[28] build a multilayer Multiscale Convolutional Neural Networks(CNN)to reduce the risk of false detection of lung nodules.Manickavasagam et al.[29]developed Computer Aided Diagnosis(CAD)software based on CNN to detect lung nodules.El-Askary et al.[30]constructed Random Forest optimization to target lung nodules.

In general,computer-aided detection of lung nodules is moving towards intelligent development,in which the research represented by deep learning framework is the focus of current research.The current algorithm research issues mainly focus on 1)how to enhance the stability of the deep learning framework.2)In the training process,how to get a more complete training effect when the number of positive samples is limited.3)How to obtain an effective feature fusion method to realize lung nodule recognition.

In response to the above problems, this paper 1)Improve the composition of the deep belief network and build a more stable structure.2)Propose a multi-view model that conforms to the principles of vision,increase the number of positive samples,and balance the number of positive and negative samples.3)Construct a lung nodule recognition algorithm based on multi-feature vector(FV)fusion.

2 Algorithm

The lung nodules are spherical-like in space, and exist as partially highlighted circles on the CT image.The diagnosis of lung nodules usually often divided into two parts: segmentation and recognition.Segmentation can obtain the suspected area of lung nodule [31] and recognition is to ensure a low missed detection rate of lung nodules while reducing the false positive rate[32].We focus on the recognition part in this paper.Under the current common deep belief network structure, we build the algorithm flow chart as shown in Fig.1.According to the principle of vision, a sample model is established from six perspectives,which can quickly present the spatial structure and improve the number of positive samples.Images with different view models are input into the improved depth belief network to get the eigenvectors.Then,a feature fusion algorithm is proposed to recognize lung nodules.

Figure 1:The algorithm flow chart

2.1 Three-Dimensional Reconstruction Algorithm

Lung nodules present a spherical shape in the lungs, which is an important feature for judging whether the area is a lung nodule or not.Therefore,it is necessary to reconstruct the suspected area in three dimensions.The Feldkamp-Davis-Kress(FDK)algorithm is the current mainstream algorithm for 3D reconstruction.The specific process is as follows:firstly,the two-dimensional projection data is weighted, then the weighted projection data of different projection angles are filtered, and finally the weighted back projection reconstruction along the ray direction is carried out.

The key step of FDK algorithm is filtering.The Shepp-Logan filter function[33]is usually used as follows:

whereNis the filter width.Based on the morphological characteristics of three-dimensional lung nodules,a smooth function is constructed to reduce noise and interference from other tissues.

whereSis an adjustable parameter.

When the voxel size of the reconstruction matrix is greater than the width of the filter, the reconstructed image cannot fully express the high-frequency information,and high-frequency aliasing occurs.For this purpose,a truncation function is constructed:

whereSizepis the pixel number of the filter;Sizevis the pixel number of the reconstructed voxel.The time-domain Shepp-Logan filter function is transformed into the frequency domain through Fast Fourier Transform(FFT)changes,and then a new filter function is constructed based on this:

Through the above algorithm processing, the suspected lung nodule area is smoothed, but the contrast and edge features are suppressed to a certain extent.Therefore, it is proposed to design a high-frequency enhancement filter.

whereb1andp1are the parameters.Whenb1＞0,high frequency enhancement can be realized,andp1controls the frequency range of enhancement.g1refers to the angle.

On the basis ofFB1,the filter function is added iteratively,which can enhance different frequency bands.

Therefore,the final filter function can be achieved:

2.2 Improved Deep Belief Network

Deep Belief Network(DBN)is a generative model that allows the entire neural network to generate training data according to the maximum probability by training the weights between its neurons[34].This network has received widespread attention since its inception.And a series of research and application carried out on it.Bu et al.[35] constructs DBN to learn high-level features.Shen et al.[36]introduced Boltzmann machines to constrain DBN.Khatami et al.[37]firstly reduced the dimensionality of medical data and then extracted high-level features through DBN.Zhong et al.[38]improved the process of fine-tuning to reduce never responding or always responding latent factors.Lu et al.[39] introduced a reconstruction error model to modify DBN to predict the probability of cardiovascular occurrence.

Due to the initial weight matrix between the last hidden layer and the classification layer of the deep belief network is randomly generated, which causes the weight matrix do not have the discriminative ability,and the feature cannot be guaranteed to be suitable for the classification task.Thus,we improve the deep belief network model,shown as in Fig.2.

Figure 2:Improved DBN chart

The network consists Restricted Boltzmann Machine (RBM)and Latent Dirichlet Allocation(DLA).They input layer {h0}, hidden layer {h1,h2...hN}, and Label is the classification layer.The number of input layer nodes is equal to the dimension of input samples, and the number of classification layer nodes is equal to the number of categories in the input sample set.

C-type training sample set is defined asi= 1,2...C, whereNiis the number of classisamples andis thej-th sample in classi.

Latent Dirichlet Allocation(LDA)is an effective feature extraction method.Its purpose is to find the linear transformation matrixW, which maximizes the ratio of the inter-class dispersion to the intra-class dispersion,

whereWois the optimal projection matrix.Sbis the matrix of dispersion between sample classes.Swis the dispersion matrix within the sample class.So the process of solvingWois transformed into the process of solving generalized matrix:

Due to the rank limit problem of LDA,Rank(Sb)≤C-1,it shows that under Fisher criterion,onlyC-1 non-zero eigenvectors can be obtained,which does not meet the requirements.For this reason,we define a new matrix of inter class dispersion according to the two class problem:

It can be seen that Rank(Snb)≤min(Rank(X),Rank(XT))=Rank(X).For this reason,multiple discriminant projection vectors are obtained to meet the requirements of the number of nodes in the DBN classification layer.According to Eq.(9), the improved optimal optimization matrixWo=[w1,w2,...,wc]is obtained.

2.3 The Multi-View Model Fusion

Lung nodules present a sphere-like structure in space and a cross-sectional structure on CT images.Recognizing lung nodules from a single location has a higher risk of misdetection and missed detection.And when using deep learning training,it is impossible to obtain enough positive samples.Therefore,we establish a multi-angle model that conforms to the principle of vision,which increases the number of positive samples while reducing the false detection rate and the missed detection rate.Blood vessels and trachea are the main interfering reasons during the detection of lung nodules.Because CT imaging is a tomographic scan, the tubular structures of blood vessels and tracheas will be truncated, and their cross-section will also be a round shape.This is similar to the two-dimensional morphology of lung nodules,which make the detection become difficult.Therefore,it is necessary to build a multidirectional model to carry out researches.

It is high risk of false detection and missed detection to identify lung nodules only from a single location[40].When using deep learning training,we cannot get enough positive samples.Therefore,we build a multi-angle model to reduce the rate of false detection and missed detection while increasing the number of positive samples.

Because the size of lung nodules is not consistent, the lung nodules are normalized as a certain size.According to the axial,coronal and sagittal views,the lung nodule was identified by the section image.In order to increase the number of positive samples and achieve a balance between positive and negative samples,the characteristics of lung nodules were analyzed from three perspectives.However,only axial,coronal and sagittal sections cannot show the overall information of lung nodules.So we introduce the concept of perspective projection.

whereM(x,y)is the pixel value of the perspective projection image in(x,y).Pm(x,y)is the pixel value of the input image in the projection direction.This model is in accords with the principle of visual occlusion.The amount of calculation is greatly reduced compared with the 3D algorithm, while presenting the three-dimensional structure of the object.

Therefore,from the perspective of cube hexahedron,we construct a projection model to show the spatial structure.The six-view image is generated into feature vector(FV)after learning from DBN,and then the feature fusion strategy is established.Finally,the SVM classifier is connected to get the classification result.

In order to verify the performance of proposed algorithm,we build following fusion algorithms respectively,shown as in Fig.3.

TYPE1:Input the axial image,generate the feature vector,and then identify the lung nodules by SVM classifier.

TYPE2:Input the axial,coronal and sagittal images respectively(the coronal and sagittal images are generated from the axial images),and then use SVM classifier to judge the properties.It is better to choose the one with more modes for more accurate result.

TYPE3:Input one-view image,generate the feature vector,and then use SVM classifier to identify lung nodules.

TYPE4:Input three-view images,generate the feature vector,and then use SVM classifier to judge the properties.It is better to choose the one with more modes for more accurate result.

TYPE5:Input six-view images,generate the feature vector,and then use SVM classifier to judge the properties.It is better to choose the one with more modes for more accurate result.

TYPE6:Input six-view images,divide them into three groups to generate eigenvectors.Use SVM classifier to judge the properties.It is better to choose the one with more modes for more accurate result.

TYPE7:Input six-view images,generate feature vectors respectively and recognize lung nodules by SVM classifier.

Figure 3:Fusion algorithms chart

3 Experiment and Result Analysis

300 sets of lung CT data are collected from the early international lung cancer action project[18]database.The size of the test and training datasets is 1:2.This database includes lung nodules and normal lung data,in which the data of lung nodules are labeled by doctor blind labeling method,and the data set is constructed.Also, the data in the database are taken down at different time and by different equipment to ensure a diversity of data and the reliability of the algorithm.

The program of the algorithm is implemented in the WIN7 system using VS2018.The detection speed is positively correlated with the complexity of the data and the amount of data,with an average of 31 s/sequence.

According to the difference of lung nodule scale,when the radius of lung nodule is less than 15 pixels,it is a small nodule,and when it is more than 30 pixels,it is a large nodule.Thus,we will classify lung nodules smaller than 152πpixel into cubes of 322pixels,and lung nodules larger than 302πpixel into cubes of 642pixels.

3.1 Parameter Selection

For the deep learning network we use, the input image is 512 × 512.And for images that do not meet the size requirements,it is normalized to 512×512.In this paper,the proposed algorithm includes the parameters ofb1,p1,b2andp2.To evaluate the performance of the filter,we introduceAOMandAVMto evaluate the combination of different parameters[41],which can build the relationship between the three-dimensional reconstructed areaRgand the marked areaRsby physician,

whereAOMis proportional to the effect of the proposed algorithm.On the contrast,AVMis reversely proportional to the effect of the proposed algorithm.

As shown in Tab.1,whenb1,p1/=0,b2andp2=0,only the first layer filter works.Asb1andp1increase,whenb1=3 andp1=12,AOMandAVMreach the peak values.On this basis,the second layer filter is added.Whenb1=3 andp1=15,AOMandAVMreach the peak values.It shows that the proposed algorithm can suppress the background and enhance the area of lung nodules.

Table 1: The effect of different parameters

We analyzed the filter response curve,as shown in Fig.4.The traditional Shepp-Logan function can enhance the high frequency part, but the enhancement effect is limited.The similarity between the target and the background is not large enough.Through the selection of the above parameters,the proposed algorithm increases the difference between the target and the background,and reaches the peak value at the edge of lung nodules.Thus,effectiveness of the proposed algorithm is illustrated.

Figure 4:The filter response curve

3.2 Performance of Multi-View Fusion Algorithm

In order to verify the effect of different algorithms,we introduce ROC curve for measurement,as shown in Fig.5.TYPE1: The three-dimensional features of lung nodules are ignored by the feature judgment of single section image.The representativeness of the selected image will directly affect the recognition effect.TYPE2:With increase of the profile,the risk of missing detection and false detection is reduced.However,the three-dimensional features of lung nodules cannot be fully displayed in the section structure.

TYPE3: The one-view image is used as input to fuse part of three-dimensional information.The effect is better than single section image input.TYPE4, 5: With increase of the number of perspective images, more three-dimensional information is fused into the perspective images.The result of comprehensive judgment is the best, but with the increase of the number of classifiers, the speed decreases.TYPE6:Reducing the number of classifiers is helpful to improve the speed,but the grouping method will directly affect the classification results.TYPE7:In general,the lung nodule can be identified by inputting the six-view images together.When the number of classifiers is small.

Figure 5:ROC of different fusion algorithms

A multi-view aided diagnosis algorithm based on small sample of lung nodules is proposed in this paper,and the characteristics of lung nodules have been analyzed.The model has been built according to the characteristics of high brightness and round shape of lung nodules.Seven types of connection frames for experiments have been proposed and experimental results shows that the seventh has the best result.

This method can be extended to other fields,but it cannot be applied in practice directly.In order to get better results,it is necessary to analyze the characteristics of the target to be detected.

3.3 Lung Nodule Recognition Performance

In order to verify the performance of the improved deep belief network algorithm,we measure the performance of different algorithms from sensitivity(SEN),specificity(SPE),false positive fraction(FPF)[42]

As shown in Tab.2 and Fig.6,the type 7 fusion algorithm is better than other algorithms in terms of performance.PCA algorithm[7]replaced the original features with fewer features.The new features are linear combinations of the old features.These linear combinations maximize sample variance and make the new features irrelevant to each other.This method is more sensitive in training category data.LBP algorithm[8]features local gray invariance and rotation invariance,but the lung nodule pattern cannot be well expressed by only a single feature.DBN [34] algorithm does not consider the initial weight matrix between the last hidden layer and the classification layer when building the network,resulting in the weight matrix does not have discrimination ability.EDBN algorithm [38] optimizes the fine-tuning algorithm to improve the accuracy.Our algorithm improves the initialization structure of hidden layer and classification layer,and optimizes the stability of the algorithm.It achieved nice recognition effect.

The detection speed of the proposed algorithm is positively correlated with the number of lesions in the sequence.The average detection time of each sequence is less than 3 minutes, which greatly reduces the time of manual interpretation.

Table 2: Comparison table of algorithm effect

3.4 Algorithm Effect Display

We display the images of different angles of lung nodules,as shown in Fig.7.We select Normal lung nodule(Fig.7a),lung nodule with vascular adhesion(Fig.7b)and blood vessel(Fig.7c),Lung nodule and blood vessel are in the form of spheroids, but they cannot be distinguished from axial images.Lung nodules and blood vessels can be distinguished from V1-V6.The multi-view model can display the local texture information on the two-dimensional image,which makes the obtained features more abundant than the single section image.

Figure 6:ROC curves of different fusion algorithms

Figure 7:Algorithm effect display

4 Conclusion

In order to meet the needs of high accuracy and low false positive rate of computer-aided detection of lung nodules,the traditional deep belief network was improved to enhance network stability.Also,a multi-view model that conforms to the principle of visual perception is proposed to balance the number of positive and negative samples.And establish a feature fusion mechanism to realize the extraction of lung nodules.On this basis,the development of the judgment of subsequent lung nodules signs has been promoted.

Funding Statement:This work was supported by Science and Technology Rising Star of Shaanxi Youth(No.2021KJXX-61);The Open Project Program of the State Key Lab of CAD&CG,Zhejiang University(No.A2206);The China Postdoctoral Science Foundation(No.2020M683696XB);Natural Science Basic Research Plan in Shaanxi Province of China(No.2021JQ-455);Natural Science Foundation of China(No.62062003),Key Research and Development Project of Ningxia(Special projects for talents)(No.2020BEB04022); North Minzu University Research Project of Talent Introduction(No.2020KYQD08).

Conflicts of Interest:Bin Li contributed equally to this work.The authors declare that they have no conflicts of interest to report regarding the present study.