• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    Meta Ordinal Regression Forest for Medical Image Classification With Ordinal Labels

    2022-07-18 06:16:54YimingLeiHaipingZhuJunpingZhangandHongmingShan
    IEEE/CAA Journal of Automatica Sinica 2022年7期

    Yiming Lei, Haiping Zhu, Junping Zhang,,, and Hongming Shan,,

    Abstract—The performance of medical image classification has been enhanced by deep convolutional neural networks (CNNs),which are typically trained with cross-entropy (CE) loss.However, when the label presents an intrinsic ordinal property in nature, e.g., the development from benign to malignant tumor,CE loss cannot take into account such ordinal information to allow for better generalization. To improve model generalization with ordinal information, we propose a novel meta ordinal regression forest (MORF) method for medical image classification with ordinal labels, which learns the ordinal relationship through the combination of convolutional neural network and differential forest in a meta-learning framework. The merits of the proposed MORF come from the following two components: A tree-wise weighting net (TWW-Net) and a grouped feature selection (GFS)module. First, the TWW-Net assigns each tree in the forest with a specific weight that is mapped from the classification loss of the corresponding tree. Hence, all the trees possess varying weights,which is helpful for alleviating the tree-wise prediction variance.Second, the GFS module enables a dynamic forest rather than a fixed one that was previously used, allowing for random feature perturbation. During training, we alternatively optimize the parameters of the CNN backbone and TWW-Net in the metalearning framework through calculating the Hessian matrix.Experimental results on two medical image classification datasets with ordinal labels, i.e., LIDC-IDRI and Breast Ultrasound datasets, demonstrate the superior performances of our MORF method over existing state-of-the-art methods.

    I. INTRODUCTION

    MEDICAL image classification has been assisted by the deep learning technique [1]–[8] and has achieved tremendous progress in the past decade. Early detection and treatment of some diseases, such as cancers, are critical for reducing mortality. Fortunately, it is implicit in medical images that the image information across the different clinical stages exhibits an ordinal relationship, which can be used to improve model generalization. For example, computed tomographic (CT) images of lung nodule [9] are given with the malignancy scores from 1 to 5, where 1 means highly unlikely to be malignant, 3 is indeterminate, and 5 is highly likely to be malignant. The majority of existing lung nodule classification methods conduct binary classification while discarding indeterminate or unsure nodules [3], [8], [10]–[14].In other words, the unsure nodules that are between benign and malignant and cannot be classified by radiologists based on current scans become useless [15]. As shown in Fig. 1, a large number of nodules are indeterminate and then discarded in the binary classification problem. It is evident that the images with ordinal labels represent the development of the lesions, as do other diseases such as breast cancer.Considering that deep learning methods are data-hungry and that these medical images differ from natural images in fewer discriminative patterns, leveraging the ordinal relationship among limited medical data for training deep learning models is becoming an important topic.

    Generally, most medical image classification methods work by feeding medical images into convolutional neural networks(CNNs) and updating the parameters of the CNNs based on cross-entropy (CE) loss. However, CE loss is inferior for fitting the ordinal distribution of labels. Therefore, ordinal regression-based methods have been explored for medical image classification with ordinal labels [15]–[17]. A simple solution is to construct a series of binary classification problems and evaluate the cumulative probabilities of all binary classifiers [16], [17]. However, those binary classifiers are trained separately, which ignores the ordinal relationship.Recently, the unsure data model (UDM) [15], neural-stick breaking (NSB) [18], unimodal method [19], and soft ordinal label (SORD) [20] have been proposed to improve ordinal regression performance based on rectified label space or the probability calculation of the output of the CNNs.

    Fig. 1. The histogram of averaged malignant scores of nodules in LIDC-IDRI dataset. The red dashed lines split the nodules into three groups; i.e., benign(≤ 2.5), unsure (2 .5 ~3.5), and malignant (≥ 3.5).

    Fig. 2. The proposed MORF framework. Left: The deep ordinal regression forest with random construction of forest, which is followed by the TWW-Net.Right: The deep ordinal regression forest followed by the TWW-Net, where the forest is constructed from the GFS. During the meta train stage, the GFS features are used to guide the update of θ. The MORF framework involves three parts of parameters: θ (CNN), π (leaf nodes, i.e., the ordinal distributions), and ?(TWW-Net). Note that the parameter θ? on the right side is the first-order derivative calculated via (8). And θ? is used to obtain the TWW-Net parameter?(u+1)through (9).

    Another method of ordinal regression without changing target distributions is based on the combination of random forest and CNNs, which has been evaluated to successfully estimate human age using facial images [4], [5], [21]. These models regard the largest probability among all the dimensions of the learned tree-wise distributions as the final prediction. To incorporate the global ordinal relationship with forest-based methods, Zhuet al. proposed the convolutional ordinal regression forest (CORF) to allow the forest to predict the ordinal distributions [22]. However, these forest-based methods suffer from the following two drawbacks: 1) The compositions of all trees depend on the random selection of split nodes from the feature vector of the fully-connected (FC)layer, and the structure of the constructed forest is fixed at the very beginning of training, leading to poor generalization due to the lack of the random perturbation of features, as suggested in [23]; and 2) There exists the tree-wise prediction variance (tree-variance) because the final prediction of the forest is the average of the results obtained by all trees, i.e., all trees share the same weights and contribute equally to the final prediction.

    To address the aforementioned problems, we propose a meta ordinal regression forest (MORF) for medical image classification with ordinal labels. Fig. 2 shows the overall framework of MORF including three parts: A CNN backbone parameterized byθto extract feature representation from the input medical image, a tree-wise weighting network (TWWNet) parameterized by?to learn the tree-wise weights for reduced tree-wise prediction variance, and a grouped feature selection (GFS) module to construct the dynamic forest that is equipped with random feature perturbation. We adopt the meta-learning framework to optimize the parametersθand?alternatively [24], [25]. The way we use meta-learning is similar to those in [24], [25], which calculates the second derivatives, i.e., Hessian matrix, via meta data or meta tasks.The main difference is that our meta data are selected features via the GFS module rather than the original images; the meta data in [24] are a subset of the validation set that contains equal numbers of images of all classes, which may be infeasible in medical cases because some classes have fewer samples. With meta-learning optimization, CNN parameters can be updated with the guide of the dynamic forest, which can achieve better generalizability.

    The main contributions of this paper are summarized as follows.

    1) We propose a meta ordinal regression forest (MORF) for medical image classification with ordinal labels, which enables the forest on top of the CNNs to maintain the random perturbation of features. The MORF comprises a CNN backbone, a TWW-Net, and a GFS module.

    2) TWW-Net assigns each tree in the forest with a specific weight and alleviates the tree-wise variance that exists in the previous deep forest-related methods. Furthermore, we provide a theoretical analysis on the weighting scheme of TWW-Net and demonstrate how the meta data can guide the learning of the backbone network.

    3) The GFS module only works in the meta training stage for generating the dynamic forest that incorporates the feature random perturbation. Combined with the TWW-Net, the final trained model can be further enhanced through the randomness of the dynamic forest.

    4) The experimental results on two medical image classification datasets with ordinal labels demonstrate the superior performance of MORF over existing methods involving classification and ordinal regression. Furthermore, we also verify that our MORF can enhance the benign-malignant classification when leveraging the unsure data into the training set on the LIDC-IDRI dataset.

    We note that this work extends our previous conference paper [26] with the following major improvements:

    1) We further explore the relationship between our reconstructed dynamic forest and the random forest, which unveils that our MORF has stronger generalizability than the previous deep forest methods due to its retained random perturbation during training.

    2) We improve GFS module by using random selection without replacement, where the resulting dimension of FC output should be equal to the number of split nodes in a forest.This improvement can avoid selecting unused elements during meta-training stage and achieve further performance gain.

    3) We provide the detailed training algorithm and the theoretical analysis on the meta weighting scheme of TWWNet.

    4) For the LIDC-IDRI dataset, we demonstrate that the binary classification on the benign and malignant classes can be improved by MORF. Most importantly, when we added the unsure data into the training set, the binary classification results on the test set are improved further.

    5) We conduct extra experiments on a new breast ultrasound images (BUSI) dataset [27] to evaluate the performance of the methods.

    II. RELATED WORK

    In this section, we review the related work on the following three aspects: 1) CNN-based methods for medical image classification with emphasis on lung nodule classification and breast cancer diagnosis, 2) ordinal regression methods, and 3)meta-learning methods.

    A. CNN-Based Medical Image Classification

    Medical image classification, such as lung nodule and breast tumor classification, has benefited from advanced CNN architectures and learning strategies. In this paper, we focus on medical image classification with ordinal labels.

    Lung Nodule Classification:Liuet al. combined statistical features and artificial neural networks to detect lung nodules in full-size CT images [28]. Shenet al. applied a single column network to classify lung nodule images with different sizes [12]. Douet al. [13] explored an ensemble of subnetworks, each of which has a specific convolutional kernel size. Caoet al. trained two 3-D networks on original data and augmented data, and combined them for lung nodule detection [29]. However, 3-D networks are difficult to train with limited medical data [30], [31]. Another kind of method combines low-level and high-level features that come from UNet-like network architectures [8], [32]–[36], which can avoid information loss through the concatenation of different features.

    Breast Cancer Diagnosis:Breast cancer diagnosis has also been enhanced by deep learning [2], [18], [37]–[45]. In [41],[46], the authors conducted the CNN-based image fusion,feature fusion, and classifier fusion methods to classify breast tumors in ultrasound (US) images. Wuet al. explored the binary classification (benign and malignant) of breast cancer based on the proposed view-wise, image-wise, breast-wise,and joint learning approaches [43]. Akselrod-Ballinet al.combined CNN learned features with the electronic health records to improve the early detection of breast cancer using mammograms [44]. Dhungelet al. applied a pretrained CNN model in the classification of breast cancer and verified the effectiveness of deep features against traditionally handcrafted features [45]. Hagoset al. incorporated symmetry patches with the symmetric locations of positive tumor patches to help improve breast cancer classification performance [46].

    All the methods we reviewed above did not take into account the progression of the diseases that implies the intrinsic ordinal relationship among the classes. In this paper,we use widely-used backbone networks such as VGG [47] and ResNet [1] to verify the effectiveness of our method in exploring ordinal relationship.

    B. Ordinal Regression

    Ordinal regression is a classical problem that predicts ordinal labels [16], [17], [48], such as facial age [22], aesthetic image classification [49], and medical image malignancy ranking [15], [18], [19]. Beckham and Pal enforced each element of the FC output to obey unimodal distributions such as Poisson and Binomial [19]. The unimodal method surpassed the normal cross-entropy baseline. Neural stickbreaking (NSB) was proposed in [18], whose output is a(N?1)- dimensional vector representingN?1 boundaries whereNis the number of classes. NSB guaranteed that the cumulative probabilities would monotonically decrease. The unsure data model (UDM) is a strategy-driven method and focuses more on the fact that normal samples and disease samples should be classified with high precision and high recall, respectively [15]. The UDM incorporates some additional parameters associated with techniques like ordinal regression, class imbalance, and their proposed strategies.Although the UDM outperforms the unimodal method and NSB, it requires more effort to tune the model to obtain the optimal additional parameters. Soft ordinal label (SORD)converted the ground truth labels of all classes in a soft manner by penalizing the distance between the true rank and the label value [20].

    The methods discussed above contain a deep neural network(DNN) backbone followed by a modified classifier (FC layer)except for the SORD method. This pipeline cannot avoid the redundant use of the FC feature and may lead to overfitting.Recently appeared deep random forest-based methods targeted this problem. The deep neural decision forest (DNDF) defines a differentiable partition function for each split node [5].Hence, the forest can be updated jointly with the deep networks through backpropagation. The label distribution learning forest (LDLF) extended DNDF to output a predicted distribution [21]. However, the DNDF and LDLF have difficulty guaranteeing the global ordinal relationship of the predictions of leaf nodes. Convolutional ordinal regression forest(CORF) was proposed to incorporate the constraint of ordinal relation in the loss function, which enabled the output of the forest to be globally ordinal (monotonically decrease) [22].

    In this paper, our MORF further improves the efficiency of the FC feature and incorporates random perturbation of features for the forest. Moreover, our MORF enables different trees to have specific weights through the guidance of the meta data.

    C. Meta Learning

    Meta-learning is tailored for learning the meta knowledge from the predefined meta task set or meta dataset. It is widely used in few-shot learning. Model-agnostic meta-learning(MAML) [50] learns the parameter initialization from the fewshot tasks, and the new tasks only take a few training steps while achieving better generalization than fine-tuning. Jamal and Qi proposed extending the MAML to avoid overfitting on existing training tasks by proposing a maximum-entropy prior that introduces some inequality measures in the loss function[51]. Liuet al. enhanced the generalizability of the main task from the predefined auxiliary task using meta-learning [25].Meta-weight-net is a novel weighting network to address class imbalance and noisy label problems [24]. In summary, these meta-learning methods involve optimizing two groups of parameters jointly. The meta training algorithm of our MORF is similar to that of [24], but differs in its construction of the meta data and weighting behavior over the decision trees. The construction of the meta data in [24] is to select a subset of the validation set with an equal number of samples for each class for meta training, whereas the meta data in MORF are feature level.

    III. METHODS

    In this section, we first formulate the problem of ordinal regression forest for medical images. Then, we introduce the meta ordinal regression forest (MORF) framework in order of the training objective, TWW-Net, and GFS module. Finally,we present the meta-learning optimization algorithm and the corresponding theoretical analysis.

    A. Problem Formulation

    Ordinal regression solves the problem that the data belonging to different classes have an ordered label, which implies that an intrinsic ordinal relationship exists among the data. It learns a mapping functionh:X →у , where X represents the input space and у is the ordinal label space. Here,у={y1,y2,...,yC} has the ordinal relationshipy1≤y2≤···≤yC, whereCis the number of classes. In this study,yc∈у denotes the stage of the progression of diseases; taking the lung nodule classification [9] as an example,Cequals 3 andy1,y2, andy3represent benign, unsure and malignant,respectively.

    To solve this kind of ordinal classification problem, the givenlabely∈уcanbeconvertedtoanordinaldistribution label,i.e., a one-dimensional vectord=(d1,d2,...,dC?1)T∈D[22], wheredc=1ify>yc, otherwisedc=0. Practically,we will obtain an accuratedfor a given image, and thedshouldmaintaina monotonicallydecreasingpropertyacross all theelements. Therefore, we imposea constraintd1≥d2≥···≥dC?1ondduring training [22]. Under the framework of the ordinal regression forest (ORF) [4], [21], [22], the ordinal labeldcis given by the leaf nodes in the ORF, and the probability of the given samplexfalling into thel-th leaf node is defined as

    When we obtain the probabilityp(l|x;θ), the output of one tree can be defined as a mappingg:X →D,

    Fig. 3. The proposed grouped feature selection (GFS) assigning each tree with features of different groups. Note that the number of groups (colored boxes) in GFS equals the number of split nodes in one decision tree.

    where T denotes one decision tree, and L a set of leaf nodes.πl(wèi)holds the ordinal distribution of thel-th leaf node; i.e.,πl(wèi)=(,,...,)T.Inthispaper, theparameterπcan be updated jointlywiththatofthebackbone networkthrough back-propagation, which has been illustrated in [4], [21], [22].Then, the final prediction of the forest is the average of outputs of all trees

    whereTis the total number of trees. Here, all the trees contribute equally to the final prediction as well as that in previous deep forest methods such as the CORF [22], whereas in our MORF model, we assign each tree a specific learned weight.

    B. Meta Ordinal Regression Forests

    The total framework of the MORF contains a CNN with an FC layer as backbone network parameterized byθ, a TWWNet parameterized by?, and the leaf nodes parameterized byπ. Note that theπis updated according to [22].

    1)Objective Function:As mentioned above, all trees in ORF and CORF are assigned the same weights, which can increase the inevitable tree-wise prediction variance. To cope with this drawback, we propose multiplying the tree-wise losses with specific weights. Therefore, the gradients ofθ,backpropagated from the losses of all the trees, can be affected by the weights ωt. Therefore, our training objective function is defined as

    whereNis the number of training images,denotes the classification loss generated by (2), and ωtrepresents the specific weight for thet-th tree that is learned by the TWWNet, which will be subsequently introduced. Different from[24], (4) imposes the weights on the different trees w.r.t. one training samplei, rather than on the different samples.

    2)Tree-Wise Weighting Network:Here, we introduce the TWW-Net that is used for learning the weights ωtin (4).Similarly to the meta-weight-net [24], TWW-Net is practically implemented as a group of multilayer perceptrons (MLPs),Vt,and each MLP acts as a weight learning function for a specific tree because of the universal approximation property of MLPs[52]. In Fig. 2 , we can see that thet-th tree generates a classification lossRt, thenRtwill be fed into the corresponding weighting netVt, and finallyVtoutputs the weightwtfor thet-th tree. This process can be formulated as

    Therefore, TWW-Net is composed ofTweighting netVt,whereTis the number of trees. Here we use?={?1,?2,...,?T}to represent the set of parameters of TWW-Net, and they are updated together which will be described in Section III-C.Through (5), a TWW-Net can assign different weights to different trees.

    Combined with (4), the training objective function can be modified as follows:

    whereStrdenotes the training set.

    3)Grouped Feature Selection(GFS):Although we have incorporated the tree-wise weights in the training objective function, the structure of the forest is still fixed. Therefore, we introduce the GFS module to construct the dynamic forest with random feature perturbation in this section. Then, we explore the relationship between the GFS and random forest.

    As shown in Fig. 3, the GFS first ranks all the activation values of the final FC feature vector. Then it splits the ranked elements into N groups (denoted by different colors), where Nequals the number of split nodes in one decision tree. Both the elements inside and outside of one group are in descending order, and each tree randomly selects its own nodes across all the groups. Hence, one tree contains the features globally across the FC feature, and it retains the local random perturbation of the feature that is critical for the random forest. After repeating these procedures on all the trees, the dynamic forest is constructed. Note that the final trained model of MORF is also equipped with a forest that is fixed from the beginning of training, and the GFS module only works in the training stage and has no impact on the forest at the time of inference. Note that, the number of groups corresponds to the number of split nodes in one tree, i.e., the number of groups increases along with an increase of tree depth.

    Relationship Between GFS and Random Forest:Random forest (RF) is a classical ensemble learning method, which benefits from base learners that have feature and data random perturbations. Specifically, feature perturbation means that each node in the decision trees is the most discriminative attribute in a subset of its whole attribute set [23]. Data perturbation is satisfied for all the deep forest-based methods that trained over shuffled mini-batches, however, feature perturbation occurs because of the fixed forest structure [4],[21], [22]. In Fig. 3(b), we can see that each split node is randomly selected from its own feature set. Although, in Fig. 3(a), all the nodes in the forest are also obtained through random selection within their subsets (indicated by different colors) of the FC feature, it differs from the RF in that the nodes in different trees share the same subsets, i.e., the GFSbased forest also maintains the node-wise feature random perturbation. Therefore, the MORF with GFS possesses the merit of randomness with respect to all the split nodes, and this advantage does not exist for previous methods [4], [5],[21], [22].

    C. Optimization via Meta-Learning

    When we obtain a dynamic forest, we expect it to guide the update of the CNN with a fixed forest. Moreover, from (6) we observe that the objective function involves two parts of parameters,θand?, andθis a function of?, so we customize a meta-learning framework enabling the meta data to guide the learning of the target model. Here, the GFS selected features are regarded as the meta data, which is different from those in [24].

    To obtain the optimal θ?, we need to obtain the optimal ??.Therefore, we optimize?by minimizing the following objective function:

    whereMis the number of meta data. This objective function indicates that?is updated based on the optimal backbone parameter θ?.

    First, we take the derivative of (6) with respect toθ

    whereαis the learning rate forθ. For simplicity, we omit the parametersπand the datasetsStrandSmetain the above equations. The superscript (u) denotes theu-th iteration.Therefore, θ? in (8) represents the weights obtained through the first order derivative ?θ(θ). Then, we can use θ? to update the parameters?

    To go a step further, we derive (9) and obtain the following equation:

    where

    stands for the similarity between two gradients?the gradient of thei-th training data computed on training lossand the gradientofthemeanvalueofthemini-batchmetadata calculated on metaloss.Thisenforcesthegradientofthe feature of training data to approach that of meta data generated from GFS. Hence, the behavior of each tree is guided by the meta gradient and is consistent with other trees.Consequently, the predictions of different trees in our MORF are consistent, i.e., have lower variance, which guarantees a more stable prediction.

    Algorithm 1 Training Algorithm of MORF

    After we obtain the updated TWW-Net parameters ?(u+1),the update rule ofθcan be defined as

    In (9), the tree-wise lossRt(,π;Smeta) is calculated via meta data, and the meta data are feature level. During the trainingprocedure, wefirstobtainthe first-order derivativein(8)bytaking theimagexasinput, andthe forestis constructed based on this forward process as shown on the left side of Fig. 2. Then, we fixin (8) and take the imagexas input, and the forest here is reconstructed through our GFS module as shown on the right side of Fig. 2. Once again, the globally and locally selected features via GFS are the meta data in our method. This retains the structural variability and the random feature perturbation of the forest. Simultaneously,our training scheme can guide the behavior of learning from training data to approach that of learning from the GFS generated meta data. The details of the training procedure are shown in Algorithm 1.

    Note that the TWW-Net (?) only works in the meta training stage, and it does not affect the prediction in inference. In other words, the trained model contains parametersθandπ,which are required during inference.

    IV. EXPERIMENTAL SETUP AND RESULTS

    In this section, we present the experimental setup and evaluate the proposed MORF method on the LIDC-IDRI [9]and BUSI [27] datasets. The experiments reported the average results through five randomly independent split folds, and for the accuracies, we reported the average values with standard deviations.

    A. Data Preparation

    1)LIDC-IDRI Dataset for Lung Nodule Classification:LIDC-IDRI is a publicly available dataset for pulmonary nodule classification or detection, which involves 1010 patients. Some representative cases are shown in Fig. 4. All the nodules were labeled by four radiologists, and each nodule was rated with a score from 1 to 5, indicating malignant progression of the nodule. In our experiments, we cropped the region of interest (ROI) with a square shape of a doubled equivalent diameter at the corresponding center of a nodule.The averaged score of a nodule was used as the ground-truth label during training. Note that the averaged scores also range from 1 to 5, and we regard a nodule with an average score between 2.5 and 3.5 as the unsure nodule, benign and malignant nodules are those with scores that are lower than 2.5 and higher than 3.5, respectively [15]. In each plane, all the CT volumes were preprocessed to have 1 mm spacing in each plane. Finally, we obtain the training and testing data by cropping the 32×32×32 volume ROIs located at the annotated centers.

    2)BUSI Dataset for Breast Cancer Classification:The BUSI dataset can be used for ultrasound image-based breast cancer classification and segmentation, which contains 780 images of three classes: 133 normal, 487 benign, and 210 malignant images. Some representative cases are shown in Fig. 4. We first resized the original 2-D images into the same sizes 128×128, and then conducted the data augmentations,including flipping and adding random Gaussian noise, for the training set of BUSI. Finally, the training and test sets contain 1872 images and 156 images for each fold, respectively.

    B. Implementation Details

    Fig. 4. Some examples in the BUSI [27] and LIDC-IDRI [9] datasets. For the LIDC-IDRI dataset, we provide four examples each class.

    1)Network Architecture:We applied ResNet-18, ResNet-34, and VGG-16 [1], [47] as backbone networks to compare our MORF with other methods. Because the scales of the two datasets are relatively small, we use the 2-D version of the backbone networks to avoid the huge number of parameters in 3-D networks. Therefore, the input of the model, the 32×32×32 volumes, can be treated as 32×32 patches with 32 channels each, and the corresponding number of channels of the first layer is set as 32. For our MORF, the output dimension of the final FC layer equals the number of split nodes in a forest due to the GFS using random selection without replacement, and for CORF, it is set as 256.

    2)Hyperparameter Setting:The learning rates for the LIDC-IDR and the BUSI datasets are 0.001 and 0.00005,respectively, and are decayed by 0.1 every 120 epochs (150 epochs in total); the sizes of a mini-batch size are 16, and the weight decay values for the Adam optimizer are 0.0001 and 0.00005, respectively [53]. The loss functions used in the MORF, CORF, and each tree-wise loss during meta training are the standard CE loss [54]–[56].

    The number of trees for the forest is 4 and the tree depth is 3. In practice, the TWW-Net contains several MLPs, where he number of MLPs equals the number of trees. For the LIDC-IDRI dataset, we evaluated whether or not the unsure nodules were used for training or testing. Therefore, in Tables I and II, we use the symbol Train(n1) -Test(n2) to represent that here aren1classes of data for training, andn2classes for esting. The values ofn1andn2are 3 (with unsure data) or 2(without unsure data). All of our experiments are implemented with the PyTorch [57] framework and trained with an NVIDIA GTX 2080 Ti GPU.

    C. Training With Unsure Data for Lung Nodule Classification

    In this section, we focus on the standard 3-class classification of lung nodules. Following [15], we also care more about the recall of malignant lesions and the precision of benign lesions, which fits more appropriately with the clinical diagnosis.

    In Table I, we illustrate the results of the CE loss-based methods, the related ordinal regression methods, and our MORF as well as its conference version denoted as MORFc.When using different backbones, the MORF achieves the best accuracies and F1 scores of malignant and benign. For all the backbones, the MORF also maintains the higher recall of malignant and precision of benign. Under the meaning of the clinical diagnosis, the MORF is better able to reduce the missing diagnosis rate, i.e., there will be fewer unsure and malignant nodules diagnosed as benign, and fewer malignant nodules missing a diagnosis. Most importantly, MORF obtains the best precision for malignancy, demonstrating a lower misdiagnosis rate.

    TABLE I CLASSIFICATION RESULTS ON TEST SET OF TRAIN(3)-TEST(3) ON LIDC-IDRI DATASET. THE VALUES WITH UNDERLINES INDICATE THE BEST RESULTS WHILE LESS IMPORTANT IN THE CLINICAL DIAGNOSIS [15]. MORFC IS THE CONFERENCE VERSION OF THE PROPOSED MORF [26]

    Under the setting of Train(3)-Test(2) where training data include all unsure data, the feature space will be more complicated compared with the binary classification setting.However, the left part of Table II shows that the MORF significantly outperforms the other methods significantly on all measured metrics. We emphasize that the ordinal relationship of the data is critical to ordinal regression which can be regarded as a fine-grained classification, and the accurate feature representation determines the ability of the final classifier. Although both MORF and CORF [22] consider the global ordinal relationship, the fixed forest of the CORF degrades its performance in that the random feature perturbation is omitted. The MORF with reconstructed forest via the GFS module enables the update of the parameterθto be affected by the feature randomness, hence leading to a significant gain. Therefore, this experiment verifies the robustness of MORF against the influence of plugging unsure samples into the training data. Most importantly, the results of MORF on the left side of Table II are slightly better than those on the right side. This indicates that using the unsure class is helpful for improving the classification of the other two classes, i.e., the unsure nodules act as a boundary between the malignant and benign. Especially for ResNet-18/34, the recalls of malignant and the precisions of benign of Train(3)-Test(2) are higher than those of Train(2)-Test(2). VGG-16 achieves comparable results under these two settings, and we attribute this phenomenon to the different feature spaces learned by ResNet and VGG-16.

    When comparing the unsure class with existing methods,Table I shows both MORF and MORFcachieve better results than other methods in terms of recall rather than precision.Similarly to the importance of recall of malignant [15], the higher recall of the unsure class provides us with significant insights that there will be fewer unsure nodules likely being classified as benign or malignant. Although the unsure class contains mixed benign and malignant samples, one should not miss any malignant samples in unsure class. That is, the unsure class should be similar to malignant class that the recall is relatively more important than the precision.Therefore, MORF and MORFcare helpful for further diagnosis of nodules, such as biopsy. Consequently, MORF is more suitable for real clinical circumstance while recomm-ending more accurate diagnosis of follow-ups. It is noticed that this superiority does not hold in the UDM method [15].On the other hand, for a certain backbone, both the MORF and MORFcoutperform the CE loss-based counterpart, which exhibits their effectiveness on exploiting ordinal relationship.

    TABLE II CLASSIFICATION RESULTS ON TEST SETS OF TRAIN(3)-TEST(2) AND TRAIN(2)-TEST(2) ON LIDC-IDRI DATASET. THE VALUES WITH UNDERLINES INDICATE THE BEST RESULTS WHILE LESS IMPORTANT IN THE CLINICAL DIAGNOSIS [15]. IN THIS TABLE, P., R., AND F1 ARE ABBREVIATIONS OF PRECISION, RECALL, AND F1 SCORE, RESPECTIVELY. B. IS SHORT FOR BACKBONE. MORFC IS THE CONFERENCE VERSION OF THE PROPOSED MORF [26]

    D. Training Without Unsure Data for Lung Nodule Classification

    To verify the effectiveness of MORF on binary classification, we compare the results of all methods training without unsure data. The CE loss under Train(2)-Test(2) in Table II(right) is the conventional binary classifier whose output dimension is 2, and this is different from that of Train(3)-Test(2) whose output dimension is 3. It is clear that the MORF also achieves the best overall accuracy, precision of benign,and recall of malignant using different backbones.

    Through the comparison in Table II, we can see that the unsure data largely affect the generalizability of the compared methods. There are no severe fluctuations in the performance of MORF under the two settings, indicating that MORF is able to distinguish the samples with ordinal labels regardless of whether the ordinal margin is large (without unsure) or small (with unsure).

    Here, we would like to clarify why the performance in Table I is much lower than that seen in Table II. This is due to the imperfect performance of recognizing the unsure samples,so that it becomes unavoidable to encounter the classification errors of all classes.

    E. Classification Results on BUSI Dataset

    In Table III, we illustrate the results of all the methods on the BUSI dataset. For the benign and malignant classes, we also focused more on the precision of benign and recall of malignant. Differently from the LIDC-IDRI dataset, we can see from Fig. 4 that the first order in the BUSI dataset is the normal class, which does not contain nodules. Therefore, the benign class occupies different positions in the orders of the two datasets. Interestingly, our MORF also retains the best precisions of benign and malignant recalls as shown in Table III.This demonstrates the discriminative ability of MORF in recognizing nodules of different orders without the influence of the normal class that does not include nodules.

    Clinically, false positives of the normal class indicate that benign or malignant nodules are falsely classified as normal,which will result in an increase of missing diagnosis; in contrast, false negatives of the normal class will cause an increase in misdiagnosis. Since the precision and recall correlate with false positives and the false negative, here we suggest that the precision and recall of the normal class have equal importance weights. The results of compared methodsin Table III show that they are prone to preferring precision or recall of the normal class. For example, the UDM obtains a precision of 1.000 when using ResNet-18 while the corresponding recall is 0.259; the NSB achieves similar results with a large margin between precision and recall. However, the MORF has relatively balanced precisions and recalls, and it also maintains the best F1 scores when applying all of the backbones.

    TABLE III CLASSIFICATION RESULTS ON TEST SET OF BUSI DATASET. THE VALUES WITH UNDERLINES INDICATE THE BEST RESULTS WHILE LESS IMPORTANT IN THE CLINICAL DIAGNOSIS [15]. MORFC IS THE CONFERENCE VERSION OF THE PROPOSED MORF [26]

    F. Comparisons Between MORF and MORFc

    From Tables I?III , we can see that MORF consistently outperforms MORFcin terms of overall accuracy, which is benefited from the improved GFS using random selection without replacement. For both of the two datasets, MORF is better at identifying benign and malignant classes compared with MORFc, and this guarantees the improvements of overall performance. Recalling the essential difference between MORF and MORFcthat GFS in MORF uses random selection without replacement, as a result, MORF makes more efficient use of FC output vector while MORFccould be affected by selection of unused elements in FC output vector. Therefore,the results suggest that GFS without replacement will be more conducive to distinctive feature learning that identifies benign and malignant.

    G. Tree-Wise Variance Reduction

    Fig. 5. Some prediction results of MORF and CORF under Train(3)-Test(2) setting. The y axis represents the prediction results: 1, 2 and 3 represent benign,unsure and malignant. The GT denotes ground truth. The black vertical line on each bar represents the variance over the predictions of all trees. Green box contains some representative nodules, and red box include some failure cases. The subfigures represent different classes of nodules: (a), (b), (c), (d), (g); and (i)are malignant; (d), (e), (f), and (h) are benign.

    In Fig. 5, we also provide some failure cases obtained by MORF and CORF, i.e., Figs. 5(g)?5(i). Fig. 5(g) is malignant and its real malignant score is 3.75 which is referred to Fig. 1.However, both MORF and CORF make incorrect predictions,and this is due to blur edges or shapes of the nodule. Fig. 5(h)is a benign nodule in our study with malignant score 2.5 which is an upper bound of score range of benign class. We can see that MORF is prone to classifying it as unsure, while CORF obtains predictions of benign. Fig. 5(i) is a malignant nodule with malignant score 3.5, i.e., the lower bound of malignant, and the predictions are incorrect obtained by the two methods. We conclude that MORF and CORF can be confused by the malignant or benign nodules whose malignant scores are close to unsure class, and MORF prefers predictions of unsure for them. This phenomenon also reflects that MORF is more suitable for real clinical circumstance that requires more nodules surrounding the unsure class for further diagnosis.

    H. Ablation Study

    Here, we evaluated the effectiveness of the GFS module and TWW-Net based on CORF. The experiments were conducted on the LIDC-IDRI dataset under the Train(2)-Test(2) setting using the ResNet-18 backbone. Then, we evaluated the effects of the number and depth of trees in MORF. Finally, we conducted a significance analysis between MORF and other methods through predictions on the LIDC-IDRI testing set(Train(3)-Test(3)).

    1)GFS Module:To verify the random feature perturbation enforced by the GFS, we added the GFS to CORF termed CORF+GFS. The forest of the CORF is fixed during training and inference. In contrast, the CORF+GFS enables the forest structure to be dynamic during training only, and the training process does not include the meta train stage. That is to say,

    the CORF is equipped with random feature perturbations.From Table IV we can see that the CORF+GFS achieves better performances than the vanilla CORF. This indicates that the training of the CORF benefits from the GFS in that the GFS generated forest endows the target model (θ) with the generalizability increased by the random feature perturbation.However, the drawback of all trees sharing the same weights is not yet resolved. In addition, we observe that the precision of benign and the recall of malignant of CORF+GFS are worse than those of CORF, which can be explained as follows: GFS is specially designed to improve the metatraining of the proposed MORF while CORF does not have meta training, as a result, the GFS shall compromise the performance of CORF as expected.

    TABLE IV EFFECTIVENESS EVALUATION OF GFS AND TWW-NET ON THE LIDC-IDRI DATASET, UNDER THE TRAIN(2)-TEST(2) SETTING USING RESNET-18

    2)TWW-Net:The CORF+TWW-Net in Table IV is tailored for evaluating the TWW-Net without the GFS, i.e., the structure of the forest is also fixed, and the training process includes the meta train stage. Table IV shows that CORF+TWW-Net performs worse than the CORF. This is due to that the training data and the meta data are the same and consequently, the two terms of the multiplication inGi jas shown in (10) are the same. Therefore, theGi jis almost at the orientation of the largest gradient, and this phenomenon happens equally to all training samples. So we argue that in(10), CORF+TWW-Net could accelerate the update of?, and hence, may trigger the overfitting of TWW-Net. In other words, the update ofθis not guided by the meta data.Consequently, the meta weighting scheme of TWW-Net should be driven by the model generalizability gain from GFS.For MORF (i.e., CORF + GFS + TWW-Net), the update of parametersθcan be guided by GFS generated features,therefore,Gi jinvolves gradients of GFS features, thenGijwill slow down the update of?according to (10). Hence, the combination of GFS and meta training with TWW-Net achieves trade-off between updating the parametersθand?.

    Fig. 6. Classification accuracies with varying values of the depth of trees and number of trees on (a) LIDC-IDRI and (b) BUSI datasets.

    3)Number and Depth of Trees:Here, we further discuss the effects of the number and depth of trees. We fix one of them and evaluate the settings with various values of the other. The backbone network is VGG-16. Fig. 6 shows the performances influenced by these two factors on the LIDC-IDRI under the setting of Train(3)-Test(3) and BUSI datasets.

    In Fig. 6, we can see that the setting with the number of trees being 4 and the depth of trees being 3 achieves the best performances for both of the datasets. Lower or higher values of number and depth will decrease the performances. If the number of trees is fixed, there will be more nodes in a forest which requires FC output vector to have higher dimension,i.e., more elements. Then the parameters of the framework begin to increase, and this will affect the performance. If the depth of trees is fixed, we observe that when the number is small, i.e., 2 or 3, MORF achieves lower performances on two datasets. This indicates that fewer ensembled trees can affect the capability of the MORF, then hinder the performance improvement. In contrast, too many trees ( >4) also results in a decrease of performance which is attributed to more MLPs for weights learning, i.e., there are more parameters to learn.

    I. Significance Analysis

    In order to show the significant differences between MORF and compared methods, we compare MORF and other methods through conducting the Wilcoxon signed-rank test[58] with respect to predicted probabilities on test set. In Table V, we can see that all thep-values obtained by MORF and baseline methods are less than the significance level 0.05.So we reject the null hypothesis that MORF possesses the same prediction distribution as baseline methods. Finally, we conclude that MORF is significantly better than the baseline methods.

    TABLE V P-VALUES BETWEEN MORF AND OTHER METHODS USING DIFFERENT BACKBONES

    V. CONCLUSIONS

    In this paper, we propose a meta ordinal regression forest,termed MORF, for improving the performances of the ordinal regression in medical imaging, such as lung nodule classification and breast cancer classification. The MORF contains a grouped feature selection module that is used to generate a dynamic forest with feature random perturbation.Another critical component of the MORF is the TWW-Net which assigns each tree with a learned weight, and this enforces the predictions of different trees to have smaller variance while maintaining stable performances. The parameters of the model are learned through the meta-learning scheme which can solve the problem of integrating two parts of parameters into one training loop, and it brings the gradients of target data and meta data to be closer. Through the experiments, we have verified that the MORF can help reduce the false positives and the false negatives of the relevant classes, which is significant to the clinical diagnosis.Moreover, we have also verified that the accurate recognition of the intermediate order can improve the classification of the other classes on both sides.

    In the future, we will consider to explore attention mechanism for achieving better generalizability of deep random forest such as in [59]. Also, we will simplify the MORF framework such as using the design of light weight network or efficient loss functions [60], [61].

    国产欧美日韩一区二区三| 欧美成人免费av一区二区三区 | 色在线成人网| 一级毛片女人18水好多| 极品少妇高潮喷水抽搐| 男女免费视频国产| 欧美成人午夜精品| 老司机福利观看| 99精国产麻豆久久婷婷| 丝瓜视频免费看黄片| 亚洲av片天天在线观看| 亚洲人成伊人成综合网2020| 丝袜美腿诱惑在线| 日韩精品免费视频一区二区三区| 美女国产高潮福利片在线看| 女人被狂操c到高潮| 女人爽到高潮嗷嗷叫在线视频| 十八禁网站免费在线| 人人妻人人澡人人看| 视频区欧美日本亚洲| 嫩草影视91久久| 99香蕉大伊视频| 91大片在线观看| www.999成人在线观看| 欧美日韩瑟瑟在线播放| 两人在一起打扑克的视频| 一本一本久久a久久精品综合妖精| 免费在线观看视频国产中文字幕亚洲| 久久这里只有精品19| 国产免费av片在线观看野外av| 国产乱人伦免费视频| 一级,二级,三级黄色视频| 亚洲av成人不卡在线观看播放网| 日本一区二区免费在线视频| 久久亚洲精品不卡| 又大又爽又粗| 夜夜夜夜夜久久久久| 欧美+亚洲+日韩+国产| 久久久久精品国产欧美久久久| 99香蕉大伊视频| 欧美黄色片欧美黄色片| 亚洲三区欧美一区| 成人18禁在线播放| 天堂中文最新版在线下载| 伊人久久大香线蕉亚洲五| 成人18禁高潮啪啪吃奶动态图| 亚洲第一青青草原| 淫妇啪啪啪对白视频| 亚洲精品自拍成人| av天堂在线播放| 欧美中文综合在线视频| 欧美午夜高清在线| 又紧又爽又黄一区二区| 国产亚洲欧美98| 亚洲中文日韩欧美视频| 亚洲熟女精品中文字幕| 少妇被粗大的猛进出69影院| 婷婷成人精品国产| 亚洲综合色网址| 欧美日韩成人在线一区二区| 多毛熟女@视频| a级片在线免费高清观看视频| xxx96com| av有码第一页| 精品卡一卡二卡四卡免费| 国产精品久久久久久人妻精品电影| 欧美日韩亚洲高清精品| 欧美久久黑人一区二区| 欧美日韩视频精品一区| 成年人免费黄色播放视频| 在线观看免费高清a一片| 性少妇av在线| 日韩欧美三级三区| 国产精品一区二区在线不卡| 黄色视频,在线免费观看| 亚洲五月色婷婷综合| 亚洲精品国产一区二区精华液| 村上凉子中文字幕在线| 黑丝袜美女国产一区| av国产精品久久久久影院| 他把我摸到了高潮在线观看| 桃红色精品国产亚洲av| av免费在线观看网站| 亚洲av熟女| 国产精品一区二区精品视频观看| 久久午夜亚洲精品久久| 人人妻,人人澡人人爽秒播| 交换朋友夫妻互换小说| 亚洲国产欧美日韩在线播放| 亚洲精品乱久久久久久| 久久99一区二区三区| 99精品欧美一区二区三区四区| 国产精品综合久久久久久久免费 | 久久久久久久久免费视频了| 亚洲国产欧美一区二区综合| 人人妻人人爽人人添夜夜欢视频| 十八禁高潮呻吟视频| 在线十欧美十亚洲十日本专区| 两个人看的免费小视频| 老司机福利观看| 免费在线观看黄色视频的| 51午夜福利影视在线观看| 99在线人妻在线中文字幕 | av天堂在线播放| 欧美亚洲日本最大视频资源| 午夜精品久久久久久毛片777| 亚洲男人天堂网一区| 亚洲av成人一区二区三| 美女 人体艺术 gogo| 午夜免费观看网址| 久久精品亚洲精品国产色婷小说| 免费久久久久久久精品成人欧美视频| 少妇裸体淫交视频免费看高清 | 最新在线观看一区二区三区| 最近最新免费中文字幕在线| 精品人妻在线不人妻| 久久影院123| 国产亚洲精品久久久久5区| 久久香蕉精品热| 老司机福利观看| 男女免费视频国产| 日韩中文字幕欧美一区二区| 亚洲avbb在线观看| 91麻豆精品激情在线观看国产 | 在线观看日韩欧美| 女人久久www免费人成看片| 国产精品影院久久| 久热爱精品视频在线9| 老鸭窝网址在线观看| 精品久久久久久久毛片微露脸| 成年动漫av网址| 草草在线视频免费看| 亚洲 国产 在线| 国语自产精品视频在线第100页| 小蜜桃在线观看免费完整版高清| 精品午夜福利视频在线观看一区| 成人性生交大片免费视频hd| 舔av片在线| 欧美黄色片欧美黄色片| 久久精品国产清高在天天线| 18禁黄网站禁片免费观看直播| 久久久久免费精品人妻一区二区| a级一级毛片免费在线观看| 老师上课跳d突然被开到最大视频 久久午夜综合久久蜜桃 | 国产真人三级小视频在线观看| 人妻久久中文字幕网| 性色av乱码一区二区三区2| 亚洲片人在线观看| netflix在线观看网站| 免费大片18禁| 久久亚洲精品不卡| 一进一出抽搐gif免费好疼| 啦啦啦韩国在线观看视频| 国产精品久久电影中文字幕| 国产69精品久久久久777片| 国产亚洲精品久久久com| 亚洲国产精品成人综合色| 丰满乱子伦码专区| 午夜福利欧美成人| 中文字幕av在线有码专区| 国产精品久久电影中文字幕| 亚洲av电影在线进入| 好看av亚洲va欧美ⅴa在| 久久久久久久亚洲中文字幕 | 日韩有码中文字幕| 国产高清有码在线观看视频| 国产精品久久久人人做人人爽| 日韩欧美国产一区二区入口| 久久精品国产亚洲av涩爱 | 午夜福利在线观看免费完整高清在 | 亚洲av美国av| 精品国内亚洲2022精品成人| 91在线精品国自产拍蜜月 | 久久久久久久久久黄片| 精品人妻一区二区三区麻豆 | 九九热线精品视视频播放| 在线观看av片永久免费下载| 国产 一区 欧美 日韩| 无人区码免费观看不卡| 日韩高清综合在线| 亚洲无线观看免费| 成人av在线播放网站| 极品教师在线免费播放| 国产高清视频在线播放一区| 成人18禁在线播放| 欧美日韩精品网址| 午夜福利成人在线免费观看| www.www免费av| 国产精品一区二区三区四区久久| 亚洲 欧美 日韩 在线 免费| 精品福利观看| av黄色大香蕉| 国产中年淑女户外野战色| 最后的刺客免费高清国语| 俄罗斯特黄特色一大片| 欧美黑人欧美精品刺激| 国产主播在线观看一区二区| 亚洲美女黄片视频| 国产精品综合久久久久久久免费| 麻豆久久精品国产亚洲av| 国产一区二区三区视频了| 国产一级毛片七仙女欲春2| 国产精品99久久99久久久不卡| 亚洲国产欧美网| 亚洲不卡免费看| 中文字幕av在线有码专区| 国产探花在线观看一区二区| 国内精品久久久久精免费| 国产成人av激情在线播放| 99久国产av精品| aaaaa片日本免费| 在线免费观看不下载黄p国产 | 欧美一级毛片孕妇| 国产欧美日韩一区二区精品| tocl精华| 俺也久久电影网| 国产精品嫩草影院av在线观看 | 国产真实伦视频高清在线观看 | 黑人欧美特级aaaaaa片| 成人18禁在线播放| 人人妻人人看人人澡| 欧美最新免费一区二区三区 | 久久久久久久午夜电影| 亚洲中文字幕一区二区三区有码在线看| 国产综合懂色| 欧美丝袜亚洲另类 | 老汉色av国产亚洲站长工具| 丰满人妻一区二区三区视频av | 88av欧美| 日本精品一区二区三区蜜桃| 女生性感内裤真人,穿戴方法视频| 国产高清视频在线观看网站| 18禁国产床啪视频网站| 99久久精品热视频| 色哟哟哟哟哟哟| 成年版毛片免费区| 高潮久久久久久久久久久不卡| e午夜精品久久久久久久| 天堂动漫精品| 日本 av在线| 夜夜看夜夜爽夜夜摸| 在线十欧美十亚洲十日本专区| 免费看光身美女| 熟女电影av网| 又粗又爽又猛毛片免费看| 午夜福利高清视频| 动漫黄色视频在线观看| 韩国av一区二区三区四区| 搞女人的毛片| 亚洲午夜理论影院| 免费av观看视频| 免费在线观看日本一区| 精品99又大又爽又粗少妇毛片 | 欧美大码av| 色尼玛亚洲综合影院| 午夜福利成人在线免费观看| 欧美中文综合在线视频| 日韩欧美在线二视频| 亚洲精品乱码久久久v下载方式 | 亚洲男人的天堂狠狠| 欧美日韩一级在线毛片| av片东京热男人的天堂| 亚洲精品456在线播放app | 中出人妻视频一区二区| 老司机午夜福利在线观看视频| 成人亚洲精品av一区二区| 日本三级黄在线观看| 国内精品久久久久精免费| 看黄色毛片网站| av天堂中文字幕网| 国产精品99久久99久久久不卡| 国产精品久久久久久久电影 | 变态另类丝袜制服| 久久久久久国产a免费观看| 88av欧美| 精华霜和精华液先用哪个| 欧美日韩瑟瑟在线播放| 欧美激情久久久久久爽电影| netflix在线观看网站| 小说图片视频综合网站| 网址你懂的国产日韩在线| 国产精品久久久久久久电影 | 男女下面进入的视频免费午夜| 日韩精品青青久久久久久| 91久久精品电影网| 啦啦啦免费观看视频1| 亚洲av美国av| 久久国产乱子伦精品免费另类| 国产免费一级a男人的天堂| 中文字幕av成人在线电影| 久久国产乱子伦精品免费另类| 黄色女人牲交| 国产精品爽爽va在线观看网站| 国产精品久久视频播放| 一进一出抽搐动态| 内地一区二区视频在线| 免费人成在线观看视频色| 老司机午夜福利在线观看视频| 国产亚洲精品久久久com| 久久天躁狠狠躁夜夜2o2o| 乱人视频在线观看| 国产黄a三级三级三级人| 国产高清视频在线观看网站| 亚洲av美国av| 国产精品 国内视频| 成人鲁丝片一二三区免费| 女人被狂操c到高潮| 91麻豆av在线| 啦啦啦韩国在线观看视频| 成人高潮视频无遮挡免费网站| 99久久久亚洲精品蜜臀av| 亚洲精品日韩av片在线观看 | 亚洲av不卡在线观看| 中亚洲国语对白在线视频| 午夜激情福利司机影院| 99热这里只有是精品50| 啪啪无遮挡十八禁网站| 亚洲av第一区精品v没综合| 男插女下体视频免费在线播放| tocl精华| 女人被狂操c到高潮| 欧美中文综合在线视频| 午夜福利成人在线免费观看| 日本三级黄在线观看| 怎么达到女性高潮| 又黄又粗又硬又大视频| 欧美不卡视频在线免费观看| 欧美+亚洲+日韩+国产| 国产高清激情床上av| 国产一区二区在线av高清观看| 99国产精品一区二区蜜桃av| 一进一出抽搐动态| 国产精品久久电影中文字幕| 99热精品在线国产| 激情在线观看视频在线高清| 日韩欧美 国产精品| 午夜a级毛片| 国产一区二区三区在线臀色熟女| 性色avwww在线观看| 成年女人毛片免费观看观看9| 99精品在免费线老司机午夜| 欧美+日韩+精品| 99视频精品全部免费 在线| 欧美又色又爽又黄视频| 色吧在线观看| 亚洲七黄色美女视频| 99riav亚洲国产免费| 欧美色欧美亚洲另类二区| 国产综合懂色| 久久久久国产精品人妻aⅴ院| 在线观看66精品国产| 人妻夜夜爽99麻豆av| 一a级毛片在线观看| 国产精品一及| 国产伦精品一区二区三区视频9 | 日本免费a在线| 久久精品影院6| 两个人视频免费观看高清| av女优亚洲男人天堂| 国产精品久久久久久久久免 | 亚洲成a人片在线一区二区| 精品人妻一区二区三区麻豆 | 亚洲av成人精品一区久久| 成年版毛片免费区| 99国产极品粉嫩在线观看| 色吧在线观看| 免费人成视频x8x8入口观看| 国产美女午夜福利| 尤物成人国产欧美一区二区三区| 中文字幕av在线有码专区| 亚洲国产高清在线一区二区三| 欧美日韩综合久久久久久 | 真实男女啪啪啪动态图| 精品久久久久久久人妻蜜臀av| 亚洲中文日韩欧美视频| 欧美精品啪啪一区二区三区| 欧美不卡视频在线免费观看| 99久久久亚洲精品蜜臀av| avwww免费| 国产黄a三级三级三级人| 亚洲av电影在线进入| 亚洲五月婷婷丁香| 欧美成狂野欧美在线观看| 国产中年淑女户外野战色| 国产成人av激情在线播放| 欧美区成人在线视频| 美女免费视频网站| 国产精品精品国产色婷婷| 免费看十八禁软件| 女人被狂操c到高潮| 免费人成在线观看视频色| 久久久久国产精品人妻aⅴ院| 一进一出抽搐gif免费好疼| avwww免费| 亚洲自拍偷在线| 美女黄网站色视频| 制服丝袜大香蕉在线| 久久久色成人| 欧美日韩亚洲国产一区二区在线观看| a级一级毛片免费在线观看| 一级黄色大片毛片| 婷婷精品国产亚洲av在线| 亚洲国产欧洲综合997久久,| 老鸭窝网址在线观看| 免费看日本二区| 亚洲无线观看免费| 亚洲乱码一区二区免费版| 久久久久久大精品| 美女cb高潮喷水在线观看| 窝窝影院91人妻| 亚洲美女黄片视频| 老鸭窝网址在线观看| 国产 一区 欧美 日韩| 国产99白浆流出| 免费电影在线观看免费观看| www.www免费av| 看片在线看免费视频| 亚洲美女黄片视频| 午夜老司机福利剧场| 成人欧美大片| 国产高潮美女av| 天堂√8在线中文| 久久久色成人| 日本黄色片子视频| 欧美中文综合在线视频| 人人妻人人看人人澡| 国产伦精品一区二区三区四那| 欧美最新免费一区二区三区 | 亚洲乱码一区二区免费版| 麻豆国产97在线/欧美| 欧美日韩一级在线毛片| 99热这里只有是精品50| 少妇的逼水好多| 亚洲 欧美 日韩 在线 免费| 在线视频色国产色| 国产精品久久久人人做人人爽| 亚洲av第一区精品v没综合| 18美女黄网站色大片免费观看| 亚洲国产欧美网| 变态另类成人亚洲欧美熟女| 久久人人精品亚洲av| 中文字幕熟女人妻在线| 欧美日韩福利视频一区二区| 国产精品电影一区二区三区| 成年女人看的毛片在线观看| 亚洲国产精品999在线| 色综合站精品国产| 成年人黄色毛片网站| 老司机深夜福利视频在线观看| 色精品久久人妻99蜜桃| 首页视频小说图片口味搜索| 欧美日韩亚洲国产一区二区在线观看| 欧美最黄视频在线播放免费| 日本黄色视频三级网站网址| 热99在线观看视频| 一区福利在线观看| 搡女人真爽免费视频火全软件 | 国产主播在线观看一区二区| 午夜福利免费观看在线| 18禁黄网站禁片午夜丰满| 性欧美人与动物交配| 岛国在线免费视频观看| 狂野欧美白嫩少妇大欣赏| 韩国av一区二区三区四区| 日韩成人在线观看一区二区三区| 亚洲美女黄片视频| 黄色视频,在线免费观看| 99久久无色码亚洲精品果冻| 午夜免费激情av| 制服人妻中文乱码| 全区人妻精品视频| 亚洲国产中文字幕在线视频| 欧美精品啪啪一区二区三区| 97人妻精品一区二区三区麻豆| 国产不卡一卡二| 久久久久久久久大av| 男人和女人高潮做爰伦理| 日本三级黄在线观看| 波野结衣二区三区在线 | 美女cb高潮喷水在线观看| 日韩欧美国产在线观看| 可以在线观看毛片的网站| 亚洲av二区三区四区| avwww免费| 亚洲欧美日韩卡通动漫| 亚洲精品乱码久久久v下载方式 | 亚洲,欧美精品.| 亚洲五月婷婷丁香| 国内精品久久久久精免费| 手机成人av网站| 亚洲国产精品sss在线观看| 国产精品自产拍在线观看55亚洲| 美女高潮的动态| 成人永久免费在线观看视频| 国产真人三级小视频在线观看| 少妇裸体淫交视频免费看高清| 在线看三级毛片| 国产熟女xx| 国内精品美女久久久久久| 国产91精品成人一区二区三区| 最近最新免费中文字幕在线| 校园春色视频在线观看| 国产精品亚洲美女久久久| 国内揄拍国产精品人妻在线| 久久亚洲真实| 精品久久久久久成人av| 亚洲最大成人中文| 午夜福利18| 在线观看美女被高潮喷水网站 | 午夜福利欧美成人| 欧美xxxx黑人xx丫x性爽| 亚洲男人的天堂狠狠| 亚洲国产高清在线一区二区三| 免费av毛片视频| 国产精品一区二区免费欧美| 午夜福利高清视频| 久久这里只有精品中国| 好男人在线观看高清免费视频| 欧美不卡视频在线免费观看| 性色av乱码一区二区三区2| 亚洲av中文字字幕乱码综合| 有码 亚洲区| 国产 一区 欧美 日韩| 国产精品av视频在线免费观看| 很黄的视频免费| 女同久久另类99精品国产91| 日韩av在线大香蕉| 18+在线观看网站| 国产伦精品一区二区三区视频9 | 桃色一区二区三区在线观看| 老汉色∧v一级毛片| 亚洲精品久久国产高清桃花| 免费看光身美女| 噜噜噜噜噜久久久久久91| 精品一区二区三区av网在线观看| 亚洲美女视频黄频| 少妇的丰满在线观看| 亚洲av成人不卡在线观看播放网| 午夜免费激情av| av天堂中文字幕网| 亚洲欧美一区二区三区黑人| 91在线观看av| 久久香蕉国产精品| 亚洲av成人精品一区久久| 欧美午夜高清在线| 国产在视频线在精品| avwww免费| 成熟少妇高潮喷水视频| 啦啦啦免费观看视频1| 又爽又黄无遮挡网站| 国产男靠女视频免费网站| 好男人在线观看高清免费视频| 国产黄片美女视频| 欧美在线一区亚洲| 欧美绝顶高潮抽搐喷水| 叶爱在线成人免费视频播放| 9191精品国产免费久久| 综合色av麻豆| 欧美高清成人免费视频www| 丰满乱子伦码专区| 国产伦精品一区二区三区四那| 老汉色av国产亚洲站长工具| 国产成人a区在线观看| 午夜老司机福利剧场| 久久久久久久久中文| 国产成年人精品一区二区| 99久久成人亚洲精品观看| 色噜噜av男人的天堂激情| 日本撒尿小便嘘嘘汇集6| 日本免费一区二区三区高清不卡| 99在线视频只有这里精品首页| 亚洲av五月六月丁香网| av视频在线观看入口| 少妇的逼水好多| 国产精品三级大全| 国产伦精品一区二区三区四那| 亚洲精品色激情综合| 天堂网av新在线| 五月玫瑰六月丁香| 夜夜躁狠狠躁天天躁| 亚洲熟妇中文字幕五十中出| 国内精品久久久久精免费| 国产欧美日韩精品亚洲av| 午夜精品一区二区三区免费看| 亚洲,欧美精品.| 99热这里只有是精品50| 波多野结衣高清作品| 亚洲av成人精品一区久久| 国产高清视频在线观看网站| 人妻丰满熟妇av一区二区三区| 宅男免费午夜| 淫妇啪啪啪对白视频| 亚洲国产欧洲综合997久久,| 久久久久精品国产欧美久久久| 男人舔女人下体高潮全视频| 亚洲人成网站在线播| 美女免费视频网站| 中出人妻视频一区二区| 搡老岳熟女国产| 老司机午夜福利在线观看视频| 亚洲乱码一区二区免费版| 夜夜夜夜夜久久久久| 国产av在哪里看| 国产视频内射| 免费看a级黄色片| 97超级碰碰碰精品色视频在线观看| 97碰自拍视频| 少妇人妻精品综合一区二区 | 色吧在线观看| 夜夜躁狠狠躁天天躁| 国产黄a三级三级三级人| 中文字幕高清在线视频| 国产精品自产拍在线观看55亚洲| 国产伦在线观看视频一区| 男女之事视频高清在线观看|