• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    Reducing parameter space for neural network training

    2020-07-01 05:13:24TongQinLingZhouDongbinXiu

    Tong Qin, Ling Zhou, Dongbin Xiu*

    Department of Mathematics, The Ohio State University, Columbus, OH 43210, USA

    Keywords:Rectified linear unit network Universal approximator Reduced space

    ABSTRACT For neural networks (NNs) with rectified linear unit (ReLU) or binary activation functions, we show that their training can be accomplished in a reduced parameter space. Specifically, the weights in each neuron can be trained on the unit sphere, as opposed to the entire space, and the threshold can be trained in a bounded interval, as opposed to the real line. We show that the NNs in the reduced parameter space are mathematically equivalent to the standard NNs with parameters in the whole space. The reduced parameter space shall facilitate the optimization procedure for the network training, as the search space becomes (much) smaller. We demonstrate the improved training performance using numerical examples.

    Interest in neural networks (NNs) has significantly increased in recent years because of the successes of deep networks in many practical applications.

    Complex and deep neural networks are known to be capable of learning very complex phenomena that are beyond the capabilities of many other traditional machine learning techniques.The amount of literature is too large to mention. Here we cite just a few review type publications [1–7].

    In an NN network, each neuron produces an output in the following form

    where the vector xinrepresents the signal from all incoming connecting neurons, w are the weights for the input, σ is the activation function, with b as its threshold. In a complex (and deep) network with a large number of neurons, the total number of the free parameters w and b can be exceedingly large. Their training thus poses a tremendous numerical challenge, as the objective function (loss function) to be optimized becomes highly non-convex and with highly complicated landscape [8].Any numerical optimization procedures can be trapped in a local minimum and produce unsatisfactory training results.

    This paper is not concerned with numerical algorithm aspect of the NN training. Instead, the purpose of this paper is to show that the training of NNs can be conducted in a reduced parameter space, thus providing any numerical optimization algorithm a smaller space to search for the parameters. This reduction applies to the type of activation functions with the following scaling property: for any α ≥0, σ (α·y)=γ(α)σ(y), where γ depends only on α. The binary activation function [9], one of the first used activation functions, satisfies this property with γ ≡1. The rectified linear unit (ReLU) [10, 11], one of the most widely used activation functions nowadays, satisfies this property with γ =α. For NNs with this type of activation functions, we show that they can be equivalently trained in a reduced parameter space. More specifically, let the length of the weights w be d. Instead of training w in Rd, they can be equivalently trained as unit vector with ∥ w∥=1, which means w ∈Sd?1, the unit sphere in Rd. Moreover, if one is interested in approximating a function defined in a compact domain D, the threshold can also be trained in a bounded interval b ∈[?XB,XB], whereas opposed to the entire real line b ∈R. It is well known that the standard NN with single hidden layer is a universal approximator, cf. Refs. [12–14]. Here we prove that our new formulation in the reduced parameter space is also a universal approximator, in the sense that its span is dense in C (Rd). We then further extend the parameter space constraints to NNs with multiple hidden layers. The major advantage of the constraints in the weights and thresholds is that they significantly reduce the search space for these parameters during training. Consequently, this eliminates a potentially large number of undesirable local minima that may cause training optimization algorithm to terminate prematurely,which is one of the reasons for unsatisfactory training results. We then present examples for function approximation to verify this numerically. Using the same network structure, optimization solver, and identical random initialization, our numerical tests show that the training results in the new reduced parameter space is notably better than those from the standard network.More importantly, the training using the reduced parameter space is much more robust against initialization.

    This paper is organized as follows. We first derive the constraints on the parameters using network with single hidden layer, while enforcing the equivalence of the network. Then we prove that the constrained NN formulation remains a universal approximator. After that, we present the constraints for NNs with multiple hidden layers. Finally, we present numerical experiments to demonstrate the improvement of the network training using the reduced parameter space. We emphasize that this paper is not concerned with any particular training algorithm.Therefore, in our numerical tests we used the most standard optimization algorithm from Matlab?. The additional constraints might add complexity to the optimization problem. Efficient optimization algorithms specifically designed for solving these constrained optimization problem will be investigated in a separate paper.

    Let us first consider the standard NN with a single hidden layer, in the context for approximating an unknown response function f :Rd→R. The NN approximation using activation function σ takes the following form,

    where wj∈Rdis the weight vector, bj∈R the threshold, cj∈R,and N is the width of the network.

    We restrict our discussion to following two activation functions. One is the ReLU,

    The other one is the binary activation function, also known as heaviside/step function,

    with σ (0)=1/2.

    We remark that these two activation functions satisfy the following scaling property. For any y ∈R and α ≥0, there exists a constant γ ≥0 such that

    where γ depends only on α but not on y. The ReLU satisfies this property with γ (α)=α, which is known as scale invariance. The binary activation function satisfies the scaling property with γ(α)≡1.

    We also list the following properties, which are important for the method we present in this paper.

    ? For the binary activation function Eq. (4), for any x ∈R,

    ? For the ReLU activation function Eq. (3), for any x ∈R and any α,

    We first show that the training of Eq. (2) can be equivalently conducted with constraint ∥ wj∥=1, i.e., unit vector. This is a straightforward result of the scaling property Eq. (5) of the activation function. It effectively reduces the search space for the weights from wj∈Rdto wj∈Sd?1, the unit sphere in Rd.

    Proposition 1. Any neural network construction Eq. (2) using the ReLU Eq. (3) or the binary Eq. (4) activation functions has an equivalent form

    Proof. Let us first assume ∥ wj∥?=0 for all j =1,2,...,N. We then have

    where γ is the factor in the scaling property Eq. (5) satisfied by both ReLU and binary activation functions. Upon defining

    we have the following equivalent form as in Eq. (8)

    Next, let us consider the case ∥ wj∥=0 for some j ∈{1,2,...,N}.The contribution of this term to the construction Eq. (2) is thus

    The proof immediately gives us another equivalent form, by combining all the constant terms from Eq. (2) into a single constant first and then explicitly including it in the expression.

    Corollary 2. Any neural network construction Eq. (2) using the ReLU Eq. (3) or the binary Eq. (4) activation functions has an equivalent form

    We now present constraints on the thresholds in Eq. (2). This is applicable when the target function f is defined on a compact domain, i.e., f :D →R , with D ?Rdbounded and closed. This is often the case in practice. We demonstrate that any NN Eq. (2)can be trained equivalently in a bounded interval for each of the thresholds. This (significantly) reduces the search space for the thresholds.

    Proposition 3. With the ReLU Eq. (3) or the binary Eq. (4) activation function, let Eq. (2) be an approximator to a function f :D →R , where D ?Rdis a bounded domain. Let

    Then, Eq. (2) has an equivalent form

    Proof. Proposition 1 establishes that Eq. (2) has an equivalent form Eq. (8)

    where the bound Eq. (10) is used.

    Let us first consider the case

    Next, let us consider the case>XB, then·x+>0 for all x ∈D. Let J ={j1,j2,...,jL}, L ≥1, be the set of terms in Eq. (8) that satisfy this condition. We then have·x+>0, for all x ∈D,and ? =1,2,...,L. We now show that the net contribution of these terms in Eq. (8) is included in the equivalent form Eq. (11).

    (1) For the binary activation function Eq. (4), the contribution of these terms to the approximation Eq. (8) is

    Again, using the relation Eq. (6), any constant can be expressed by a combination of binary activation terms with thresholds=0. Such terms are already included in Eq. (11).

    (2) For the ReLU activation Eq. (3), the contribution of these terms to Eq. (8) is

    where the last equality follows the simple property of the ReLU function σ(y)?σ(?y)=y. Using Proposition 1, the first two terms then have an equivalent form using unit weightand with zero threshold, which is included in Eq.(11). For the constant, we again invoke the relation Eq. (7) and represent it bywhereis an arbitrary unit vector and 0 <α

    We remark that the equivalent form in the reduced parameter space Eq. (11) has different number of “active” neurons than the original unrestricted case Eq. (2). The equivalence between the standard NN expression Eq. (2) and the constrained expression Eq. (11) indicates that the NN training can be conducted in a reduced parameter space. For the weights wjin each neuron,its training can be conducted in Sd?1, the d-dimensional unit sphere, as opposed to the entire space Rd. For the threshold, its training can be conducted in the bounded interval [? XB,XB], as opposed to the entire real line R. The reduction of the parameter space can eliminate many potential local minima and therefore enhance the performance of numerical optimization during the training.

    This advantage can be illustrated by the following one dimensional example. Consider an NN with one neuron N(x)=cσ(wx+b). Suppose we are given one data point ( 1,1),then we would like to fix parameters c ∈R, w ∈R and b ∈R by minimizing the following mean squared loss, i.e.,

    Global minimizers for this objective function lie in the set

    By the definition of the ReLU function, in the region

    we have J (c,w,b)≡1, which means every point in Z is a local minima.

    However, if we consider the equivalent minimization problem in the reduced parameter space, i.e.,

    The local minima set now is reduced to

    which is much smaller than Z.

    By universal approximation property, we aim to establish that the constrained formulations Eqs. (8) and (11) can approximate any continuous function. To this end, we define the following set of functions on Rd

    where Λ ∈Rdis the weight set, Θ ∈R the threshold set. We also denote ND(σ;Λ,Θ) as the same set of functions when confined in a compact domain D ?Rd.

    By following this definition, the standard NN expression and our two constrained expressions correspond to the following spaces

    where Sd?1is the unit sphere in Rdbecause=1.

    The universal approximation property for the standard unconstrained NN expression Eq. (2) has been well studied, cf.Refs. [13-17], and Ref. [12] for a survey. Here we cite the following result for N (σ;Rd,R).

    Theorem 4 (Ref. [17], Theorem 1). Let σ be a function in(R), of which the set of discontinuities has Lebesgue measure zero. Then the set N (σ;Rd,R) is dense in C (Rd), in the topology of uniform convergence on compact sets, if and only if σ is not an algebraic polynomial almost everywhere.

    We now examine the universal approximation property for the first constrained formulation Eq. (8).

    Theorem 5. Let σ be the binary function Eq. (4) or the ReLU function Eq. (3), then we have

    and the set N (σ;Sd?1,R) is dense in C (Rd), in the topology of uniform convergence on compact sets.

    Proof. Obviously, we have N (σ;Sd?1,R)?N(σ;Rd,R). By Proposition 1, any element N (x)∈N(σ;Rd,R) can be reformulated as an element N (x)∈N(σ;Sd?1,R). Therefore, we have N(σ;Rd,R)?N(σ;Sd?1,R). This concludes the first statement Eq. (17). Given the equivalence Eq. (17), the denseness result immediately follows from Theorem 4, as both the ReLU and the binary activation functions are not polynomials and are continuous everywhere except at a set of zero Lebesgue measure.

    We now examine the second constrained NN expression Eq.(11).

    Fig. 1. Numerical results for Eq. (26) with one sequence of random initialization.

    Theorem 6. Let σ be the binary Eq. (4) or the ReLU Eq. (3) activation function. Let x ∈D ?Rd, where D is closed and bounded with. Define Θ =[?XB,XB], then

    Furthermore, ND(σ;Sd?1,Θ) is dense in C (D) in the topology of uniform convergence.

    Proof. Obviously we have ND(σ;Sd?1,Θ)?ND(σ;Rd,R). On the other hand, Proposition 3 establishes that for any element N(x)∈ND(σ;Rd,R), there exists an equivalent formulation(x)∈ND(σ;Sd?1,Θ) for any x ∈D. This implies ND(σ;Rd,R)?ND(σ;Sd?1,Θ). We then have Eq. (18).

    For the denseness of ND(σ;Sd?1,Θ) in C (D), let us consider any function f ∈C(D). By the Tietze extension theorem (cf. Ref.[18]), there exists an extension F ∈C(Rd) with F (x)=f(x) for any x ∈D. Then, the denseness result of the standard unconstrained E={x ∈Rd:∥x∥≤XB} and any given ?>0, there exists N(x)∈N(σ;Rd,R) such that NN expression (Theorem 4) implies that, for the compact set

    By Proposition 3, there exists an equivalent constrained NN expression∈ND(σ;Sd?1,Θ) such that N(x)=N(x) for any x ∈D. We then immediately have, for any f ∈C(D) and any given ?>0, there exists N (x)∈ND(σ;Sd?1,Θ) such that

    Fig. 2. Numerical results for Eq. (26) with a second sequence of random initialization.

    The proof is now complete.

    We now generalize the previous result to feedforward NNs with multiple hidden layers. Let us again consider approximation of a multivariate function f :D →R , where D ?Rdis a compact subset of Rdwith

    Consider a feedforward NN with M layers, M ≥3, where m=1 is the input layer and m =M the output layer. Let Jm,m=1,2,...,M be the number of neurons in each layer. Obviously, we have J1=d and JM=1 in our case. Let y(m)∈RJmbe the output of the neurons in the m-th layer. Then, by following the notation from Ref. [5], we can write

    where W(m?1)∈RJm?1×Jmis the weight matrix and b(m)is the threshold vector. In component form, the output of the j-th neuron in the m-th layer is

    The derivation for the constraints on the weights vectorcan be generalized directly from the single-layer case and we have the following weight constraints,

    The constraints on the thresholddepend on the bounds of the output from the previous layer y(m?1).

    For the ReLU activation function Eq. (3), we derive from Eq.(20) that for m =2,3,...,M,

    If the domain D is bounded and withthen the constraints on the threshold can be recursively derived.Starting from ∥ y(1)∥=∥x∥∈[?XB,XB] and b(j2)∈[?XB,XB], we then have

    For the binary activation function Eq. (4), we derive from Eq.(20) that for m =2,3,...,M ?1,

    Then, the bounds for the thresholds are

    We present numerical examples to demonstrate the properties of the constrained NN training. We focus exclusively on the ReLU activation function Eq. (3) due to its overwhelming popularity in practice.

    Given a set of training samples, the weights and thresholds are trained by minimizing the following mean squared error (MSE)

    We conduct the training using the standard unconstrained NN formulation Eq. (2) and our new constrained Eq. (11) and compare the training results. In all tests, both formulations use exactly the same randomized initial conditions for the weights and thresholds. Since our new constrained formulation is irrespective of the numerical optimization algorithm, we use one of the most accessible optimization routines from MATLAB?, the function fminunc for unconstrained optimization and the function fmincon for constrained optimization. For the optimization options, we use the sequential quadratic programming (spg) algorithm for both the constrained and unconstrained optimization, with the same optimality tolerance 10?6and maximum iteration number 1500. It is natural to explore the specific form of the constraints in Eq. (11) to design more effective constrained optimization algorithms. This is,however, out of the scope of the current paper.

    Fig. 3. Numerical results for Eq. (26) with a third sequence of random initialization

    After training the networks, we evaluate the network approximation errors using another set of samples—a validation sample set, which consists of randomly generated points that are independent of the training sample set. Even though our discussion applies to functions in arbitrary dimension d ≥1, we present only the numerical results in d =1 and d =2 because they can be easily visualized.

    We first examine the approximation results using NNs with single hidden layer, with and without constraints.

    We first consider a one-dimensional smooth function

    The constrained formulation Eq. (11) becomes

    Due to the simple form of the weights and the domain D =[0,1],the proof of Proposition 3 also gives us the following tighter bounds for the thresholds for this specific problem,

    Fig. 4. Numerical results for Eq. (29). a Unconstrained formulation N (x) in Eq. (2). b Constrained formulation N (x) in Eq. (11).

    We approximate Eq. (26) with NNs with one hidden layer,which consists of 2 0 neurons. The size of the training data set is 200. Numerical tests were performed for different choices of random initializations. It is known that NN training performance depends on the initialization. In Figs. 1–3, we show the numerical results for three sets of different random initializations. In each set, the unconstrained NN Eq. (2), the constrained NN Eq.(27) and the specialized constrained NN with Eq. (28) use the same random sequence for initialization. We observe that the standard NN formulation without constraints (2) produces training results critically dependent on the initialization. This is widely acknowledged in the literature. On the other hand, our new constrained NN formulations are more robust and produce better results that are less sensitive to the initialization. The tighter constraint Eq. (28) performs better than the general constraint Eq. (27), which is not surprising. However, the tighter constraint is a special case for this particular problem and not available in the general case.

    We next consider two-dimensional functions. In particular,we show result for the Franke's function [19]

    Fig. 5. Numerical results for 1D function Eq. (26) with feedforward NNs of { 1,20,10,1}. From top to bottom: training results using three different random sequences for initialization. Left column: results by unconstrained NN formulation Eq. (19). Right column: results by NN formulation with constraints Eqs. (21) and (23).

    with ( x,y)∈[0,1]2. Again, we compare training results for both the standard NN without constraints Eq. (2) and our new constrained NN Eq. (8), using the same random sequence for initialization. The NNs have one hidden layer with 4 0 neurons.The size of the training set is 5 00 and that of the validation set is 1000. The numerical results are shown in Fig. 4. On the left column, the contour lines of the training results are shown, as well as those of the exact function. Here all contour lines are at the same values, from 0 to 1 with an increment of 0.1. We observe that the constrained NN formulation produces visually better result than the standard unconstrained formulation. On the right column, we plot the function value along y=0.2x.Again, the improvement of the constrained NN is visible.

    We now consider feedforward NNs with multiple hidden layers. We present results for both the standard NN without constraints Eq. (19) and the constrained ReLU NNs with the constraints Eqs. (21) and (23). We use the standard notation{J1,J2,...,JM} to denote the network structure, where Jmis the number of neurons in each layer. The hidden layers are J2,J3,...,JM?1. Again, we focus on 1D and 2D functions for ease of visualization purpose, i.e., J1=1,2.

    Fig. 6. Numerical results for 1D function Eq. (26) with feedforward NNs of { 1,10,10,10,10,1}. From top to bottom: training results using three different random sequences for initialization. Left column: results by unconstrained NN formulation Eq. (19). Right column: results by NN formulation with constraints Eqs. (21) and (23).

    We first consider the one-dimensional function Eq. (26). In Fig. 5, we show the numerical results by NNs of { 1,20,10,1}, using three different sequences of random initializations, with and without constraints. We observe that the standard NN formulation without constraints Eq. (19) produces widely different results. This is because of the potentially large number of local minima in the cost function and is not entirely surprising. On the other hand, using the exactly the same initialization, the NN formulation with constraints Eqs. (21) and (23) produces notably better results, and more importantly, is much less sensitive to the initialization. In Fig. 6, we show the results for NNs with{1,10,10,10,10,1} structure. We observe similar performance—the constrained NN produces better results and is less sensitive to initialization.

    We now consider the two-dimensional Franke's function Eq.(29). In Fig. 7, the results by NNs with { 2,20,10,1} structure are shown. In Fig. 8, the results by NNs with { 2,10,10,10,10,1} structure are shown. Both the contour lines (with exactly the same contour values: from 0 to 1 with increment 0.1) and the function value at y =0.2x are plotted, for both the unconstrained NN Eq.(19) and the constrained NN with the constraints Eqs. (21) and(23). Once again, the two cases use the same random sequence for initialization. The results show again the notably improvement of the training results by the constrained formulation.

    In this paper we presented a set of constraints on multi-layer feedforward NNs with ReLU and binary activation functions. The weights in each neuron are constrained on the unit sphere, as opposed to the entire space. This effectively reduces the number of parameters in weights by one per neuron. The threshold in each neuron is constrained to a bounded interval, as opposed to the entire real line. We prove that the constrained NN formulation is equivalent to the standard unconstrained NN formulation. The constraints on the parameters, even though may increase the complexity of the optimization problem, reduce the search space for network training and can potentially improve the training results. Our numerical examples for both single hidden layer and multiple hidden layers verify this finding.

    Fig. 7. Numerical results 2D function Eq. (29) with NNs of the structure { 2,20,10,1}. Top row: results by unconstrained NN formulation Eq.(19). Bottom row: results by constrained NN with Eqs. (21) and (23). Left column: contour plots. Right column: function cut along y =0.2x.Dashed lines are the exact function.

    Fig. 8. Numerical results 2D function Eq. (29) with NNs of the structure { 2,10,10,10,10,1}. Top row: results by unconstrained NN formulation Eq. (19). Bottom row: results by constrained NN with Eqs. (21) and (23). Left column: contour plots. Right column: function cut along y =0.2x.Dashed lines are the exact function.

    成人高潮视频无遮挡免费网站| 少妇的逼好多水| 免费久久久久久久精品成人欧美视频 | 精品人妻视频免费看| 又粗又硬又长又爽又黄的视频| 国产亚洲最大av| 王馨瑶露胸无遮挡在线观看| 中文字幕免费在线视频6| 国产伦理片在线播放av一区| 国产黄色免费在线视频| 有码 亚洲区| 国产精品久久久久久av不卡| 九九爱精品视频在线观看| 国产伦理片在线播放av一区| 哪个播放器可以免费观看大片| 日韩强制内射视频| 久久99热6这里只有精品| av专区在线播放| 九草在线视频观看| 日韩中字成人| 久久6这里有精品| 国产黄片视频在线免费观看| 超碰av人人做人人爽久久| 国产精品爽爽va在线观看网站| 日韩三级伦理在线观看| 亚洲欧洲日产国产| 欧美成人一区二区免费高清观看| 天美传媒精品一区二区| 久久综合国产亚洲精品| 超碰av人人做人人爽久久| av女优亚洲男人天堂| 嘟嘟电影网在线观看| 亚洲成人一二三区av| 青春草国产在线视频| 如何舔出高潮| 高清毛片免费看| 丝袜喷水一区| 日韩欧美一区视频在线观看 | 国产精品蜜桃在线观看| 久久久久性生活片| 日韩亚洲欧美综合| 天堂俺去俺来也www色官网| 日本黄色片子视频| 国产 精品1| 高清欧美精品videossex| 五月伊人婷婷丁香| 亚洲av.av天堂| 国产欧美另类精品又又久久亚洲欧美| 99热网站在线观看| 2018国产大陆天天弄谢| 高清av免费在线| 国产亚洲最大av| 嫩草影院入口| 天天躁日日操中文字幕| 一区二区三区免费毛片| 国内精品宾馆在线| 国产精品99久久99久久久不卡 | 免费人成在线观看视频色| 国产色婷婷99| 国产av一区二区精品久久 | 亚洲人成网站在线观看播放| 男女啪啪激烈高潮av片| 成人漫画全彩无遮挡| 国产成人aa在线观看| 99久久精品一区二区三区| 亚洲第一区二区三区不卡| 成人国产av品久久久| 欧美日韩视频精品一区| 国产av国产精品国产| 国产精品蜜桃在线观看| 精品国产露脸久久av麻豆| 日韩大片免费观看网站| 久久久久精品性色| 在线观看免费视频网站a站| 日本-黄色视频高清免费观看| 亚洲一级一片aⅴ在线观看| 午夜老司机福利剧场| 亚洲精品456在线播放app| 免费看光身美女| 日本色播在线视频| 久久97久久精品| 久久久久久人妻| 亚洲国产日韩一区二区| 七月丁香在线播放| 18禁裸乳无遮挡免费网站照片| 亚洲不卡免费看| 亚洲欧美日韩东京热| 亚洲成人中文字幕在线播放| av国产久精品久网站免费入址| 亚洲图色成人| av在线蜜桃| 国产男女内射视频| 国产69精品久久久久777片| 欧美+日韩+精品| 国产精品无大码| 亚洲,一卡二卡三卡| 国产欧美日韩一区二区三区在线 | 国产伦在线观看视频一区| 亚洲精品日韩在线中文字幕| 国产高清国产精品国产三级 | 久久精品国产自在天天线| 久久 成人 亚洲| 亚洲精品乱码久久久v下载方式| 99视频精品全部免费 在线| 日韩精品有码人妻一区| 国产亚洲精品久久久com| 夜夜爽夜夜爽视频| 自拍偷自拍亚洲精品老妇| 天堂8中文在线网| 亚洲精品456在线播放app| 色综合色国产| 麻豆国产97在线/欧美| 99热6这里只有精品| 91精品一卡2卡3卡4卡| 亚洲成人中文字幕在线播放| 大香蕉97超碰在线| 亚洲av中文字字幕乱码综合| av免费观看日本| av国产精品久久久久影院| 成年av动漫网址| 欧美日韩一区二区视频在线观看视频在线| a级毛片免费高清观看在线播放| 久久精品国产亚洲网站| h日本视频在线播放| 国产成人91sexporn| 日本一二三区视频观看| 国产毛片在线视频| 日日撸夜夜添| 亚洲精品第二区| 亚洲久久久国产精品| 26uuu在线亚洲综合色| av国产精品久久久久影院| 亚洲精品一区蜜桃| 国产免费福利视频在线观看| 最近2019中文字幕mv第一页| 国产伦在线观看视频一区| 国内揄拍国产精品人妻在线| 国语对白做爰xxxⅹ性视频网站| 在线观看一区二区三区激情| av在线观看视频网站免费| 观看美女的网站| 纯流量卡能插随身wifi吗| 波野结衣二区三区在线| 成人影院久久| 欧美3d第一页| 国产成人午夜福利电影在线观看| 我的老师免费观看完整版| 亚洲av日韩在线播放| 男女免费视频国产| 免费观看a级毛片全部| 狂野欧美激情性xxxx在线观看| 六月丁香七月| 亚洲四区av| 熟女人妻精品中文字幕| av线在线观看网站| 色5月婷婷丁香| 国产女主播在线喷水免费视频网站| 日韩制服骚丝袜av| 国产精品免费大片| av国产久精品久网站免费入址| 久久精品国产亚洲网站| 国产午夜精品一二区理论片| 国产免费一级a男人的天堂| 少妇人妻 视频| 我要看日韩黄色一级片| 成年免费大片在线观看| 波野结衣二区三区在线| 99九九线精品视频在线观看视频| 国产亚洲av片在线观看秒播厂| 我的老师免费观看完整版| 超碰av人人做人人爽久久| 97在线人人人人妻| 这个男人来自地球电影免费观看 | 久久ye,这里只有精品| 成人二区视频| 国产免费一级a男人的天堂| 在线观看美女被高潮喷水网站| 99久久人妻综合| 高清黄色对白视频在线免费看 | 啦啦啦啦在线视频资源| av.在线天堂| 日韩强制内射视频| 亚洲第一av免费看| av免费观看日本| 亚洲四区av| 欧美+日韩+精品| 亚洲天堂av无毛| 国产午夜精品久久久久久一区二区三区| 欧美老熟妇乱子伦牲交| 在线精品无人区一区二区三 | 免费人成在线观看视频色| 欧美成人一区二区免费高清观看| 18禁在线无遮挡免费观看视频| 免费看av在线观看网站| 亚洲人成网站高清观看| 久久女婷五月综合色啪小说| 亚洲怡红院男人天堂| 免费大片18禁| 亚洲精品第二区| 亚洲av中文字字幕乱码综合| 成人毛片60女人毛片免费| 国产伦精品一区二区三区视频9| 天堂8中文在线网| 国产精品久久久久久精品古装| 小蜜桃在线观看免费完整版高清| 亚洲欧美精品自产自拍| 深夜a级毛片| 嫩草影院新地址| 韩国av在线不卡| 国产伦理片在线播放av一区| 91aial.com中文字幕在线观看| 国产黄色免费在线视频| 国产精品女同一区二区软件| 大码成人一级视频| 国产高清有码在线观看视频| 亚洲图色成人| av.在线天堂| 少妇的逼好多水| 欧美丝袜亚洲另类| 久久精品国产亚洲av涩爱| 精品国产乱码久久久久久小说| 嫩草影院入口| 在线观看一区二区三区激情| 身体一侧抽搐| 亚洲精品色激情综合| 在线免费观看不下载黄p国产| 国产精品国产av在线观看| 国产精品福利在线免费观看| 国产成人免费无遮挡视频| 熟女人妻精品中文字幕| 内射极品少妇av片p| av免费观看日本| 久久ye,这里只有精品| 久久99热6这里只有精品| 国产成人freesex在线| 草草在线视频免费看| 国内精品宾馆在线| 久久毛片免费看一区二区三区| 国产精品国产三级专区第一集| 爱豆传媒免费全集在线观看| 亚洲四区av| 久久av网站| 免费观看a级毛片全部| 王馨瑶露胸无遮挡在线观看| 精品一区二区三卡| 交换朋友夫妻互换小说| 亚洲av福利一区| 久久精品国产亚洲网站| av福利片在线观看| 亚洲自偷自拍三级| 2021少妇久久久久久久久久久| 国产午夜精品一二区理论片| 亚洲激情五月婷婷啪啪| 深爱激情五月婷婷| 免费在线观看成人毛片| 日韩电影二区| 一级毛片黄色毛片免费观看视频| 久久精品久久久久久噜噜老黄| 成人高潮视频无遮挡免费网站| av国产免费在线观看| 天堂8中文在线网| 免费看日本二区| 欧美+日韩+精品| 3wmmmm亚洲av在线观看| 久久99热这里只频精品6学生| 青春草国产在线视频| 日韩,欧美,国产一区二区三区| 日韩中字成人| 欧美成人a在线观看| 婷婷色av中文字幕| 赤兔流量卡办理| 亚洲欧美一区二区三区黑人 | 男人狂女人下面高潮的视频| 亚洲性久久影院| 国产午夜精品久久久久久一区二区三区| 深爱激情五月婷婷| 国产69精品久久久久777片| 亚洲在久久综合| 亚洲欧洲国产日韩| 午夜福利视频精品| 国产精品伦人一区二区| 国产在线视频一区二区| 午夜日本视频在线| 天堂8中文在线网| 亚洲精品国产av蜜桃| 欧美+日韩+精品| 哪个播放器可以免费观看大片| 欧美成人a在线观看| 少妇精品久久久久久久| av在线观看视频网站免费| 久久韩国三级中文字幕| 国产免费又黄又爽又色| 亚洲欧美日韩无卡精品| 丰满少妇做爰视频| 日韩欧美一区视频在线观看 | 日本av免费视频播放| 久久青草综合色| 久久精品国产a三级三级三级| 亚洲av中文av极速乱| 亚洲,一卡二卡三卡| 日韩视频在线欧美| av网站免费在线观看视频| 国产亚洲91精品色在线| 少妇人妻精品综合一区二区| 日韩成人伦理影院| 欧美日韩精品成人综合77777| 春色校园在线视频观看| 观看免费一级毛片| av网站免费在线观看视频| 超碰av人人做人人爽久久| 亚洲av在线观看美女高潮| 老司机影院毛片| 七月丁香在线播放| 在线免费观看不下载黄p国产| 亚洲性久久影院| av免费在线看不卡| 丝袜脚勾引网站| 亚洲在久久综合| 久久久久久伊人网av| 国产淫片久久久久久久久| 久久久久国产网址| 亚洲精品国产成人久久av| 国产免费又黄又爽又色| 一级片'在线观看视频| 国产高清三级在线| 一本色道久久久久久精品综合| 国产爽快片一区二区三区| 亚洲天堂av无毛| 国产亚洲一区二区精品| 欧美日韩在线观看h| 在线免费观看不下载黄p国产| 91久久精品电影网| 国产美女午夜福利| 久久人妻熟女aⅴ| 欧美xxxx黑人xx丫x性爽| 亚洲欧美日韩东京热| 综合色丁香网| 人人妻人人爽人人添夜夜欢视频 | 久久国产亚洲av麻豆专区| 人人妻人人爽人人添夜夜欢视频 | 日韩欧美精品免费久久| 国产有黄有色有爽视频| 偷拍熟女少妇极品色| 小蜜桃在线观看免费完整版高清| 成人影院久久| 麻豆成人av视频| 成人影院久久| 伦精品一区二区三区| 多毛熟女@视频| 欧美97在线视频| 777米奇影视久久| 精品亚洲成a人片在线观看 | videos熟女内射| 蜜臀久久99精品久久宅男| 一区二区三区免费毛片| 97超碰精品成人国产| av免费观看日本| 国产 精品1| 久久久久久久久久成人| 欧美日韩在线观看h| 亚洲天堂av无毛| 日韩欧美精品免费久久| 看十八女毛片水多多多| 日韩成人av中文字幕在线观看| 国产免费又黄又爽又色| 中文字幕久久专区| 欧美日韩视频精品一区| 下体分泌物呈黄色| 在线播放无遮挡| 亚洲自偷自拍三级| 中文字幕免费在线视频6| av在线观看视频网站免费| 在线 av 中文字幕| 国产男女超爽视频在线观看| xxx大片免费视频| 亚洲不卡免费看| 欧美精品一区二区大全| 亚洲,一卡二卡三卡| 亚洲精品一二三| 日本色播在线视频| 国产精品久久久久成人av| 又黄又爽又刺激的免费视频.| 看非洲黑人一级黄片| 卡戴珊不雅视频在线播放| 六月丁香七月| 久久鲁丝午夜福利片| 又粗又硬又长又爽又黄的视频| 亚洲人成网站在线播| 亚洲欧美精品专区久久| 国产成人aa在线观看| 看十八女毛片水多多多| 美女xxoo啪啪120秒动态图| 久久精品熟女亚洲av麻豆精品| 99热国产这里只有精品6| 熟女人妻精品中文字幕| 精品国产一区二区三区久久久樱花 | 亚洲熟女精品中文字幕| 国产精品一区www在线观看| 黑丝袜美女国产一区| 久久影院123| 国产成人精品一,二区| 日本vs欧美在线观看视频 | 一级av片app| 日韩一本色道免费dvd| 高清欧美精品videossex| www.av在线官网国产| 夫妻午夜视频| 国产色爽女视频免费观看| 精品国产三级普通话版| 少妇 在线观看| 精品国产露脸久久av麻豆| 91在线精品国自产拍蜜月| 午夜免费观看性视频| 一区二区av电影网| 欧美丝袜亚洲另类| 日本黄大片高清| 国产av国产精品国产| 午夜视频国产福利| 日韩三级伦理在线观看| 国产亚洲一区二区精品| 人妻夜夜爽99麻豆av| 久久6这里有精品| 国产精品不卡视频一区二区| 亚洲欧洲日产国产| 国产一区有黄有色的免费视频| 精品国产三级普通话版| 国产69精品久久久久777片| av网站免费在线观看视频| 日本一二三区视频观看| 国产精品.久久久| 激情 狠狠 欧美| 久久婷婷青草| 97超视频在线观看视频| 亚洲精品国产av蜜桃| 日本色播在线视频| 麻豆乱淫一区二区| 男女啪啪激烈高潮av片| 国产人妻一区二区三区在| 女性生殖器流出的白浆| 亚洲精品久久午夜乱码| 国产av一区二区精品久久 | 国产黄色视频一区二区在线观看| 亚洲美女黄色视频免费看| 免费黄网站久久成人精品| 亚洲一级一片aⅴ在线观看| 精品一品国产午夜福利视频| 天堂8中文在线网| h日本视频在线播放| 2021少妇久久久久久久久久久| 国产乱人视频| 久久精品久久久久久久性| 99热国产这里只有精品6| 国产大屁股一区二区在线视频| 欧美精品国产亚洲| 美女中出高潮动态图| 国产精品精品国产色婷婷| 黄色一级大片看看| 国产在视频线精品| 这个男人来自地球电影免费观看 | 欧美丝袜亚洲另类| 亚洲怡红院男人天堂| 精品久久久噜噜| 日韩强制内射视频| 久久久久久久久久久免费av| 国产淫片久久久久久久久| 国产精品偷伦视频观看了| 欧美高清性xxxxhd video| av又黄又爽大尺度在线免费看| 成人综合一区亚洲| 久久热精品热| 中文字幕精品免费在线观看视频 | 观看av在线不卡| 香蕉精品网在线| 色婷婷av一区二区三区视频| 精品视频人人做人人爽| 18禁在线播放成人免费| 欧美日韩综合久久久久久| a 毛片基地| av播播在线观看一区| 久久久久久人妻| 中文精品一卡2卡3卡4更新| av专区在线播放| 久久久久久久精品精品| 亚洲国产高清在线一区二区三| 午夜激情福利司机影院| 少妇人妻 视频| 美女中出高潮动态图| 天堂8中文在线网| 欧美区成人在线视频| 狂野欧美激情性bbbbbb| 亚洲欧洲国产日韩| 精品久久久噜噜| 免费大片18禁| 亚洲国产高清在线一区二区三| 韩国av在线不卡| 日本wwww免费看| 精品一区二区三卡| 久久青草综合色| 免费观看av网站的网址| 三级国产精品片| 亚洲伊人久久精品综合| 国产又色又爽无遮挡免| 少妇猛男粗大的猛烈进出视频| 亚洲第一区二区三区不卡| 建设人人有责人人尽责人人享有的 | 99九九线精品视频在线观看视频| 色网站视频免费| 久久久久精品性色| 26uuu在线亚洲综合色| 中文字幕久久专区| 一区二区三区四区激情视频| 日本-黄色视频高清免费观看| a级毛色黄片| 欧美bdsm另类| 精品一区二区三卡| 亚洲欧美日韩卡通动漫| 寂寞人妻少妇视频99o| a级毛色黄片| 日日撸夜夜添| 亚洲国产精品999| 久久韩国三级中文字幕| 中文天堂在线官网| 99热这里只有精品一区| 中文精品一卡2卡3卡4更新| 不卡视频在线观看欧美| 久久久久网色| 26uuu在线亚洲综合色| 免费人成在线观看视频色| 91在线精品国自产拍蜜月| 大话2 男鬼变身卡| 久久久久久久大尺度免费视频| 亚洲国产精品专区欧美| 人人妻人人看人人澡| 久久午夜福利片| 天天躁夜夜躁狠狠久久av| 欧美高清性xxxxhd video| 国产精品欧美亚洲77777| 看十八女毛片水多多多| 久久久久国产精品人妻一区二区| 久久久成人免费电影| 欧美高清成人免费视频www| 丰满少妇做爰视频| 自拍偷自拍亚洲精品老妇| 国产 一区 欧美 日韩| 成人影院久久| 全区人妻精品视频| 免费看日本二区| 国产国拍精品亚洲av在线观看| 伊人久久精品亚洲午夜| 久久久久精品性色| 日本wwww免费看| 亚洲精品一二三| 亚洲欧美清纯卡通| 亚洲欧美精品专区久久| 国产色婷婷99| 91精品伊人久久大香线蕉| 久久热精品热| 97在线人人人人妻| 欧美日韩视频高清一区二区三区二| 亚洲欧美一区二区三区黑人 | 成人高潮视频无遮挡免费网站| av女优亚洲男人天堂| 精品国产三级普通话版| 日韩在线高清观看一区二区三区| 日本一二三区视频观看| 亚洲欧美日韩无卡精品| 精品视频人人做人人爽| 18禁在线无遮挡免费观看视频| 欧美亚洲 丝袜 人妻 在线| 久久精品国产a三级三级三级| 中国国产av一级| 午夜福利在线观看免费完整高清在| 国产v大片淫在线免费观看| 欧美精品国产亚洲| 三级经典国产精品| 国产精品一二三区在线看| 黑人高潮一二区| 黄色一级大片看看| 亚洲精品成人av观看孕妇| 一本色道久久久久久精品综合| 国产一区二区三区综合在线观看 | 国产亚洲5aaaaa淫片| 精品午夜福利在线看| 99久久精品一区二区三区| 久久人人爽av亚洲精品天堂 | a级一级毛片免费在线观看| 一级毛片电影观看| 另类亚洲欧美激情| 久久人人爽av亚洲精品天堂 | 一区二区三区四区激情视频| 亚洲经典国产精华液单| 最近手机中文字幕大全| 亚洲av日韩在线播放| 久久精品夜色国产| 欧美精品一区二区免费开放| 亚洲不卡免费看| 国产男人的电影天堂91| 国产精品不卡视频一区二区| 高清不卡的av网站| 成年美女黄网站色视频大全免费 | 日韩av在线免费看完整版不卡| 香蕉精品网在线| 亚洲精品乱久久久久久| 国产又色又爽无遮挡免| 欧美zozozo另类| 国产成人精品福利久久| av不卡在线播放| 亚洲精品亚洲一区二区| 成年av动漫网址| 日韩av免费高清视频| 日韩一区二区视频免费看| 综合色丁香网| 日韩av不卡免费在线播放| 少妇精品久久久久久久| 男男h啪啪无遮挡|