• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach

    2019-10-26 01:12:16ZhenLIShishengZHONGLinLIN
    CHINESE JOURNAL OF AERONAUTICS 2019年9期

    Zhen LI, Shisheng ZHONG, Lin LIN

    School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China

    KEYWORDS Aero-engine;Hybrid strategy;Maintenance policy;Optimization algorithm;Reinforcement learning

    Abstract An aero-engine maintenance policy plays a crucial role in reasonably reducing maintenance cost. An aero-engine is a type of complex equipment with long service-life. In engineering,a hybrid maintenance strategy is adopted to improve the aero-engine operational reliability. Thus,the long service-life and the hybrid maintenance strategy should be considered synchronously in aero-engine maintenance policy optimization. This paper proposes an aero-engine life-cycle maintenance policy optimization algorithm that synchronously considers the long service-life and the hybrid maintenance strategy. The reinforcement learning approach was adopted to illustrate the optimization framework, in which maintenance policy optimization was formulated as a Markov decision process. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the maintenance policy. Compared with traditional aero-engine maintenance policy optimization methods,the long service-life and the hybrid maintenance strategy could be addressed synchronously by the proposed algorithm. Two numerical experiments and algorithm analyses were performed to illustrate the optimization algorithm in detail.

    1. Introduction

    An aero-engine, which is composed of mechanic-electrichydraulic coupling systems,is the power plant of an aircraft.1,2It has been reported that more than 30% of aircraft mechanical problems are related to aero-engines, and the aero-engine maintenance cost contributes to about 30% of an airline's direct operating cost.3A maintenance optimization method provides an available way to reduce the maintenance cost reasonably.4In general, excessive maintenance is costly, while insufficient maintenance may lead to disasters.Thus,a maintenance policy plays a crucial role in balancing the maintenance cost and operational reliability.5However, it is not an easy work to optimize an aero-engine maintenance policy manually,especially taking the long service-life and the hybrid maintenance strategy into consideration synchronously.

    In engineering,a hybrid maintenance strategy is adopted to improve the civil aero-engine operational reliability.Strategies of Condition-Based Maintenance (CBM), Hard-time Maintenance (HM), and failure Corrective Maintenance (CM) are included in the hybrid maintenance strategy.6As aero-engine performance deterioration is inevitable,7gas path performance parameters are monitored for CBM. The Life Limit Part(LLP) should be replaced before its life limitation,8and HM is adopted. Moreover, CM is performed when an aeroengine is in the random failure state.Thus,the hybrid maintenance strategy should be considered in aero-engine maintenance policy optimization. However, few existing aero-engine maintenance optimization methods are able to address the hybrid strategy. At the same time, an aero-engine is a type of equipment with long service-life,9which should be synchronously considered in maintenance policy optimization.To the best of our knowledge, this is the first paper to investigate an aero-engine maintenance policy optimization method that can synchronously address the hybrid maintenance strategy and long service-life.

    In traditional aero-engine maintenance optimization methods, maintenance intervals are used to decide when to repair an aero-engine, and maintenance work-scopes indicate how to carry out maintenance actions. Maintenance intervals and work-scopes can be obtained by traditional separate models.For example, LLP replacement intervals and performance recovery intervals can be optimized separately by current models.10-12Based on maintenance intervals, maintenance workscopes are obtained by traditional decision-making models.13,14Based on traditional optimization methods, an aeroengine maintenance decision support system was proposed by Fu et al.,15in which maintenance interval and work-scope optimization models were presented. An optimization method for reliability-centered maintenance was proposed by Crocker and Kumar,16in which the concepts of soft life and hard life were used to optimize a military aero-engine maintenance policy.A multi-objective evolutionary algorithm was also adopted to solve the aero-engine maintenance scheduling problem,17taking the module exchange into consideration. To optimize an aero-engine maintenance policy, traditional optimization methods would become extremely complicated when the long service-life and the hybrid maintenance strategy are considered synchronously.18

    In general, machine learning methods include supervised learning, unsupervised learning, and reinforcement learning,19and the reinforcement learning method has attracted increasing interests in solving decision-making problems.20Reinforcement learning represents a machine learning method in which an agent learns how to behave through action rewards.Different from widely used supervised learning methods, there is no presentation of input and output pairs in reinforcement learning. In reinforcement learning, an agent chooses an available action according to the environment on each decision epoch.The chosen action changes the environment, along with a reward to the agent. The objective of the agent is to find the action collection, whose reward is maximal in the long run.

    Reinforcement learning methods have been adopted in energy system charging policy optimization,21,22energy system and distributed system schedule determination,23,24multiple robotic task optimization,25demand response optimization,26,27robust control optimization,28multiple satellites task planning,29et al. Although reinforcement learning methods have been successfully applied, they have not aroused much attention in aero-engine maintenance policy optimization. Reinforcement learning does provide a more appropriate way for aero-engine life-cycle maintenance policy optimization. Thus, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on reinforcement learning. The main contributions of this paper are as follows:

    (1) An aero-engine life-cycle maintenance policy optimization algorithm is proposed, which can synchronously address the aero-engine long service-life and the hybrid maintenance strategy.

    (2) To address the hybrid strategy, the aero-engine state is represented by a multi-dimensional state space.

    (3) Reinforcement learning is adopted to illustrate the maintenance policy optimization. In the reinforcement learning framework, maintenance policy optimization is formulated as a discrete Markov Decision Process(MDP), and the Gauss-Seidel value iteration algorithm is adopted to optimize the maintenance policy.

    The remainder of this paper is organized as follows.Section 2 introduces aero-engine maintenance policy optimization and the reinforcement learning approach, and the deficiencies of traditional optimization methods are analyzed.In Section 3, the proposed aero-engine life-cycle maintenance policy optimization algorithm is described in detail. In Section 4, two simulation experiments and algorithm analysis are used to illustrate the proposed optimization algorithm.Conclusions and future work are discussed in Section 5.

    2. Aero-engine maintenance policy optimization and reinforcement learning

    2.1. Aero-engine maintenance policy optimization

    A maintenance policy indicates when and how to repair an aero-engine. Traditionally, maintenance intervals indicate when to repair an aero-engine, which are obtained by traditional separate optimization models, addressing CBM, HM and,CM strategies separately.In traditional methods,maintenance work-scopes indicate how to repair an aero-engine,which are optimized based on the outputs of interval optimization models. In summary, there are several deficiencies in traditional aero-engine maintenance optimization methods, as follows:

    (1) In traditional methods,maintenance intervals and workscopes are optimized by separate models, and workscopes are optimized based on interval optimization results. Thus, the interactions of maintenance intervals and work-scopes are neglected.Moreover,interval optimization errors would be propagated to work-scope optimization.

    (2) Because the hybrid maintenance strategy is adopted for civil aero-engines, hybrid strategies should be addressed synchronously in optimization. However, traditional optimization methods address hybrid strategies separately, and interactions of hybrid strategies are neglected.

    (3) It is difficult for traditional optimization methods to address the aero-engine long-service life and the hybrid maintenance strategy synchronously.

    (4) Definite optimization results are obtained by traditional optimization methods. Due to random factors, definite optimization results may be poorly applicable in engineering.

    To deal with the aforementioned deficiencies of traditional methods, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on the reinforcement learning approach. Taking the place of maintenance intervals and work-scopes in traditional methods, the maintenance policy indicates when and how to repair an aero-engine.The proposed optimization algorithm is able to address the hybrid maintenance strategy and long service-life synchronously.

    To address the hybrid maintenance strategy, a multidimensional space is adopted to represent the aero-engine state.The maintenance strategies of CBM and CM are formulated as an MDP,and imperfect repair and random factors are all considered in state transition. Aero-engine maintenance policy optimization is illustrated by the reinforcement learning approach, and the Gauss-Seidel value iteration algorithm is adopted to seek for the optimal maintenance policy. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm makes it available to optimize the lifecycle maintenance policy. A comparison between traditional optimization methods and the proposed optimization algorithm is shown in Fig. 1.

    In Fig.1,traditional aero-engine maintenance optimization methods are shown on the left.Traditional methods are constituted by separate models,including an LLP interval optimization model, a performance interval optimization model, and a work-scope optimization model.LLP replacement and performance recovery intervals are obtained by separate optimization models. Based on the LLP replacement, performance recovery, and corrective maintenance intervals, the maintenance work-scope is optimized. The proposed aero-engine life-cycle maintenance policy optimization algorithm is shown on the right in Fig. 1. Based on the reinforcement learning framework, the strategies of HM, CM, and CBM are addressed synchronously by the proposed optimization algorithm. The traditional separate optimization models of the LLP replacement interval, the performance recovery interval,and the maintenance work-scope are replaced by the proposed optimization algorithm.

    2.2. Reinforcement learning approach

    Reinforcement learning is a machine learning method,which is widely used to solve multi-step, sequential-decision problems.Different from supervised learning, no pre-specified model is required in reinforcement learning. In aero-engine life-cycle maintenance policy optimization, few historical data is available for training the pre-specified mode. Thus, reinforcement learning provides a more appropriate way for aero-engine life-cycle maintenance policy optimization. Meanwhile, the aero-engine long service-life and the hybrid maintenance strategy can be addressed synchronously by reinforcement learning.

    In reinforcement learning, an agent takes on the work of optimizing an aero-engine maintenance policy. The agent is able to respond to dynamically changing aero-engine states through ongoing learning methods.30In aero-engine maintenance policy optimization, aero-engine states are represented by a multi-dimensional state space. To optimize the maintenance policy, the agent chooses a maintenance action according to the aero-engine state. The aero-engine state is changed by the chosen maintenance action,along with the maintenance cost, as shown in Fig. 2. The optimal objective of the agent is to find the maintenance action collection, whose total cost is the minimum in the long run.

    Fig. 2 Schematic diagram of reinforcement learning.

    Value iteration is a reinforcement learning algorithm that is widely adopted in solving decision-making problems. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm provides an appropriate way to optimize an aero-engine life-cycle maintenance policy. According to the Gauss-Seidel value iteration algorithm, an agent would run multiple episodes for the purpose of exploring and finding the optimal policy.The learning process is conducted for a sufficient number of iterations,and the total cost of each iteration is recorded.The minimum total cost is represented as Q-value,which is updated every iteration, and the Bellman equation is adopted as the updating mechanism in the Gauss-Seidel value iteration algorithm. The convergence of value iterative methods has been widely proven.31Thus, based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm.

    3. Aero-engine life-cycle maintenance policy optimization algorithm

    The reinforcement learning approach is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm, in which the hybrid maintenance strategy and long service-life are considered synchronously.In the reinforcement learning framework, the aero-engine state, maintenance actions, state transition matrices, and maintenance costs should be determined. To address the hybrid maintenance strategy, a multi-dimensional state space is adopted to represent the aero-engine state, taking performance, LLP, and random failure states into consideration synchronously. The optimal objective of the proposed optimization algorithm is to obtain a maintenance policy whose total cost is the minimum in the long run. The reinforcement learning algorithm of Gauss-Seidel value iteration is adopted, addressing longterm optimization. In this section, the proposed optimization algorithm is described in detail.sub-state Yjdenotes the performance state, and Zkdenotes the random failure state.

    (1) LLP state

    In engineering,an LLP must be replaced before its life limitation,and the HM strategy is adopted for LLP maintenance.Traditionally, the flight-cycle or flight-hour is adopted to represent the LLP state. For example, a fan shaft is an LLP of a CFM56-5B aero-engine, and its life limitation is 30,000 flightcycles. That is, a fan shaft must be replaced before 30,000 flight-cycles. For convenience, the LLP state is represented by discrete time increments.32In the proposed optimization algorithm, the LLP state is represented by several state levels,denoted as {Ti|i=0,1,2,...}, where T0denotes the all-new state; when m <n, state Tnis ‘‘older” than state Tm. It is defined that an LLP is in state Tn, when tn-1<tllp≤tn; tn-1and tnare boundary values; tllpis an LLP real state, measured by flight-cycle or flight-hour.

    (2) Performance state

    As the CBM strategy is adopted in aero-engine maintenance, the aero-engine performance state should also be considered in the proposed optimization algorithm. Most of the in-service aero-engines are equipped with condition monitoring systems, and performance parameters are sent to ground by the aircraft communication addressing and reporting system in quasi real-time. Traditionally, the aero-engine performance state is assessed by performance parameters.33,34In engineering, the Exhaust Gas Temperature Margin (EGTM)is adopted as an aero-engine performance indicator.35The EGTM is defined as the margin temperature between exhaust gas temperature and red-line temperature. The aero-engine performance degrades as it operates, presented as EGTM declining.36,37When the EGTM is close to the limitation, performance recovery maintenance should be performed. The 1000 flight-cycles EGTM parameters of a CFM56-5B aeroengine are shown in Fig. 3.

    As EGTM parameters are typical time series,38it would make the algorithm extremely complicated when EGTM time series are adopted directly.For convenience,the CBM strategy is formulated as a discrete MDP in the reinforcement learning framework. Thus, the aero-engine performance state is represented by several levels, denoted as {Di|i=0,1,2,...}, where D0denotes the all-new performance state; when m <n, state Dnis worse than state Dm. It is defined that the performance state is Dn, when dn-1<dper≤dn, in which dn-1and dnare

    3.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state is changed by performed actions. As the hybrid maintenance strategy is considered in the proposed optimization algorithm,a multi-dimensional state space is adopted to represent the aero-engine state, in which performance, LLP, and random failure states are considered synchronously. The multidimensional state space is represented by S={Xi,Yj,Zk,...},where Xi,Yj,Zk,... denote the sub-states in the multidimensional state space. Each sub-state denotes one considered factor, for example, sub-state Xidenotes the LLP state,the boundary values;dperis the EGTM real value,measured by centigrade. Thus, in the proposed optimization algorithm, the aero-engine performance state is indicated by the performance levels.

    Fig. 3 1000 flight-cycles EGTM of a CFM56-5B aero-engine.

    (3) Random failure state

    Although an aero-engine is with high reliability, it is subjected to random failures in practice. When an aero-engine is in the random failure state, CM should be performed to drive it back to the working state. Unlike the LLP or performance state, the random failure state is represented by two levels,denoted as {F0,F(xiàn)1}, where F0denotes the working state and F1denotes the random failure state.

    From the above,when LLP,performance,and random failure states are all considered, the aero-engine state is represented by a multi-dimensional state space, denoted as

    where Tidenotes the LLP state, Djdenotes the performance state, and Fkdenotes the random failure state.

    3.2. Maintenance actions

    In reinforcement learning, maintenance actions are performed on every decision epoch. The decision epoch is denoted as{Eii=| 0,1,2,...,m,...}, and state Siis regarded as the status between epochs Ei-1and Ei. As LLP, performance, and random failure states are all considered,LLP replacement actions,performance recovery actions, and failure corrective actions should be determined on each decision epoch.

    LLP replacement actions denote maintenance actions of replacing an aero-engine LLP. When an LLP replacement action is performed, the definite LLP is replaced, and the LLP state is changed.LLP replacement actions are represented by {Arep,i|i=0,1,2,...}, where Arep,0denotes no LLP replaced and Arep,m(m≠0) denotes LLP m replaced. When Arep,mis performed, the LLP m state is changed to the all-new state.

    Performance recovery actions denote maintenance actions of recovering aero-engine performance. When a performance recovery action is performed, the performance state is recovered by a definite level, and the performance state is changed.Performance recovery actions are represented by{Arec,j|j=0,1,2,...}, where Arec,0denotes no performance recovery action performed and Arec,m(m≠0)denotes the action of recovering m performance levels.

    Failure corrective actions denote maintenance actions of making a failure-state aero-engine back to the running state.When the aero-engine is trapped in the random failure state,a failure corrective action should be performed.Failure corrective actions are represented by Acor={Acor,0,Acor,1}, where Acor,0denotes no corrective maintenance performed and Acor,1denotes corrective maintenance performed.

    From the above, aero-engine maintenance actions are represented by

    Because an aero-engine does not operate during the maintenance process, maintenance actions are assumed to be ‘‘instantaneous” in the proposed optimization algorithm.39

    3.3. State transition

    In the reinforcement learning framework,the aero-engine state is changed by performed maintenance actions.Thus,LLP,performance, and random failure state transitions are illustrated as follows.

    (1) LLP state transition

    In engineering,the LLP state is measured by the flight-cycle or flight-hour,which increases directly as an aero-engine operates. The LLP state would be recovered to the all-new state when an LLP replacement action is performed. Thus, the LLP state transfers directly, without uncertainty.

    When action Arep,0is performed,LLP state Tiwould transfer to the definite state Ti+1,that is,p(Ti+1|Ti,Arep,0)=1.When action Arep,1is performed, LLP state Tiwould transfer to T0,that is, p(T0|Ti,Arep,1)=1. A schematic diagram of LLP state transition is shown in Fig. 4.

    (2) Performance state transition

    Aero-engine performance deterioration is inevitable in engineering. As the imperfect maintenance concept and random factors are considered in the proposed optimization algorithm,the CBM strategy is formulated as a discrete MDP. Probability matrices are adopted in performance state transition.

    (

    Because the maintenance concept of ‘‘a(chǎn)s good as new” has been proven to be far from the truth,40a more realistic concept of imperfect repair is adopted for performance recovery actions.41,42That is,the performance state cannot be recovered to the all-new state by any maintenance actions, and the performance state would transfer according to the transition probability matrices. When action Arec,m(m >0) is performed,the performance state would transfer from Dito Di-maccording to the probability matrix [p(Di-m|Di,Arec,m,i-m >0)]. A schematic diagram of performance state transition is shown in Fig. 5.

    The performance state transition probability matrices can be calculated by survival analysis based on the Weibull distribution.43,44

    Fig. 4 Schematic diagram of LLP state transition.

    Fig. 5 Schematic diagram of performance state transition.

    (3) Random failure state transition

    As an aero-engine may be in the random fault state occasionally, the CM strategy is also formulated as a discrete MDP. The random failure state would transfer according to probability matrices.

    In the proposed optimization algorithm, an aero-engine may fall into the random failure state F1by probability p(F1|Dj,F(xiàn)0;Acor,0). To be more realistic, it is assumed that the random failure probability is lower when the aero-engine is in a better performance state. Thus, the random failure probability p(F1|Dj,F(xiàn)0) is related to the performance state Dj.When the aero-engine is in the random failure state,corrective maintenance should be performed to drive it to the working state. It is assumed that the corrective maintenance is completely efficient, and the transition probability of the corrective maintenance is represented as p(F0|Dj,F(xiàn)1;Acor,1)=1.A schematic diagram of random failure state transition is shown in Fig. 6.

    From the above, the state transition matrix on Eiis represented by p(Si+1|Si,Ai), where Si+1denotes the aero-engine state on Ei+1, Sidenotes the aero-engine state on Ei, and Aidenotes the performed maintenance action on Ei. Different sub-states would transfer according to different modes, as illustrated above.

    3.4. Total cost and optimization

    In reinforcement learning, an agent chooses maintenance actions according to action costs.44,45In the proposed optimization algorithm, the optimal objective is to obtain a maintenance policy whose total cost is the minimum in the long run.The maintenance cost on decision epoch Ekis calculated by

    where ckdenotes the maintenance cost on decision epoch Ek;Cope,kdenotes the operating cost; Crep,k,Crec,k, Ccor,k, and Cotherdenote the LLP replacement cost, the performance recovery cost,the corrective maintenance cost,and other costs,respectively.

    Fig. 6 Schematic diagram of random failure state transition.

    In general, a good-performance-state aero-engine would have a better economic efficiency and a lower random failure rate. Thus, Cope,kis determined by the performance state.When an LLP replacement action is performed, material and replacement costs are both counted in Crep,k(Arep,k). LLP replacement costs vary from different LLP replacement actions. The performance recovery cost is represented by Crec,k(Arec,k), and when m >n, Crec,i(Arec,m)>Crec,i(Arec,n).The corrective maintenance cost Ccor,kis counted when a corrective maintenance is performed.

    As the life-cycle maintenance policy is optimized by the proposed optimization algorithm, the future maintenance cost should be counted in optimization. In reinforcement learning,a discount factor is adopted to address long-term optimization.Thus,the optimal objective of the proposed optimization algorithm is denoted as

    where C denotes the discounted future cost, and γ(γ ∈[0,1])denotes the discount factor, representing the relative impact of future action costs.

    In reinforcement learning, when a larger γ is adopted,future action costs would leave a greater impact on maintenance action selection.That is,when γ=0,the optimized policy is shortsighted,and the maintenance action is chosen by the current cost; when γ=1, all the future actions are considered in action selection,which would bring a heavy calculation burden. Thus, a balance between future costs and the calculation burden should be determined. Thus, the discount factor γ should be set as γ ∈(0,1), for example, γ=0.9.

    As the Gauss-Seidel value iteration algorithm is an effective reinforcement learning algorithm, which is widely used in policy optimization, it is adopted to seek for the maintenance action collection whose discounted long-term cost is the minimum.

    4. Numerical experiments of maintenance policy optimization

    Two numerical experiments were used to illustrate the proposed aero-engine life-cycle maintenance policy optimization algorithm in detail. According to the reinforcement learning framework,determinations of aero-engine states,maintenance actions, state transition matrices, and total cost matrices are described firstly.As traditional methods are unable to address the long service-life and the hybrid maintenance strategy synchronously,they were not adopted as the benchmark methods.The reinforcement learning algorithm of Gauss-Seidel value iteration was adopted in the experiments.

    4.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state should be determined firstly. As the multi-dimensional state space was adopted to represent the aero-engine state, performance, LLP, and random failure states were all considered in the first numerical experiment.

    The EGTM was adopted to represent the aero-engine performance state. For convenience, the EGTM time series were divided into several levels. In a sample fleet, there were three performance recovery actions, including minimal repair,medium repair, and overhaul repair. According to the performance recovery assumptions in Section 4.2,there should be at least five performance levels to fully illustrate the three performance recovery actions. In reinforcement learning, as more performance levels would make the optimization algorithm more complicated, five levels were able to present the aeroengine performance state. In the numerical experiment, the performance state was divided into five levels, denoted as{D1,D2,D3,D4,D5}, from good to bad, where D5denoted the worst performance level. Besides, the all-new performance state was denoted as D0.

    Because performance and random failure states were both transferred by probabilities, the random failure state was regarded as a specific‘‘performance state”.Thus,in the numerical experiment, performance and random failure states were represented by one state-space dimension, denoted as{D0,D1,D2,D3,D4,D5,F(xiàn)}.

    Although there were several LLPs in an aero-engine, for convenience,LLPs with the same life limitation were regarded as the same LLP type. In the first numerical experiment, one LLP type was taken into account,and the LLP state was measured by flight-cycles. Referring to the adopted performance state levels,the LLP state was divided into five levels,denoted as {T1,T2,T3,T4,T5}. Besides, T0denoted the all-new state.Thus, the aero-engine state was represented by a twodimensional state space, denoted as

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state F; Tj(j=0,1,2,3,4,5)denoted the LLP state. The performance and random failure states were denoted as the first dimension; the LLP state was denoted as the second dimension.

    Fig. 7 Schematic diagrams of performance recovery action effects on performance states.

    4.2. Maintenance actions

    In reinforcement learning, the aero-engine state was changed by performed maintenance actions, and maintenance actions should be determined according to the aero-engine state.Firstly, according to traditional and practical matters,46two performance recovery assumptions were made in the numerical experiment as follows:

    (1) An aero-engine could not be recovered to the all-new performance state by any performance recovery actions.

    (2) No maintenance action should be performed when the aero-engine was in the worst performance state.

    According to the sample fleet operation, three actions of minimal repair, medium repair, and overhaul repair were included in performance recovery actions. Thus, three performance recovery actions were adopted in the numerical experiment, denoted as Arec={Arec,1,Arec,2,Arec,3}, where Arec-h1was defined as the action to recover the performance state from Dxto Dx-1, when x-1 >0, or to keep in the performance state Dx,when x=1;Arec,2was defined as the action to recover the performance state from Dxto Dx-2, when x-2 >0, or to recover the performance state from Dxto D1,when 1 ≤x ≤2;Arec,3was defined as the action to recover the performance state from Dxto Dx-3, when x-3 >0, or to recover the performance state from Dxto D1, when 1 ≤x ≤3. Besides, Arec,0denoted no performance recovery action was performed.In the reinforcement learning framework, Fig. 7 shows performance recovery action effects on performance states.

    In the numerical experiment, the random failure state was regarded as a specific ‘‘performance state”. Thus, it was assumed that performance recovery actions could drive the random failure aero-engine back to the working state.

    As one LLP type was considered,LLP replacement actions were denoted as Arep={Arep,0,Arep,1}. Fig. 8 shows the LLP replacement action effect on LLP states.

    From the above, maintenance actions in the numerical experiment were represented by

    4.3. State transition

    Fig. 8 Schematic diagram of the LLP replacement action effect on LLP states.

    Based on the aforementioned methods, performance state Diand transition probability matrix P0were obtained from the sample fleet, as shown in

    Transition matrices of Arec,1, Arec,2, and Arec,3were P1, P2,and P3, denoted as

    where nllpdenoted the number of LLP types, and nrecadenoted the number of performance recovery actions.

    4.4. Maintenance policy optimization

    Based on the aforementioned methods, aero-engine states,maintenance actions, and state transition matrices were all determined. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy. The flow diagram of the proposed aero-engine maintenance policy optimization algorithm is shown in Fig. 9.

    Table 2 Transition probability matrix of an LLP replacement action.

    Fig. 9 Flow diagram of maintenance policy optimization.

    Because real cost data was unavailable, a hypothetical cost matrix was adopted in the numerical experiment. In engineering,the cost matrix may change according to the actual maintenance cost, and it would not distort the analysis of simulation results. In the numerical experiment, the LLP replacement cost Cllpwas assumed to be 350, while the performance recovery costs Crep,1, Crep,2, and Crep,3were assumed to be 300, 350, and 500. As an aero-engine has long service-life,future maintenance actions should be fully considered in lifecycle maintenance policy optimization.Thus,a larger discount factor γ=0.9 was adopted.In contrast to the Jacobi value iteration algorithm, the Gauss-Seidel value iteration algorithm has a faster convergence ability. Thus, in the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy.

    As the aero-engine state was represented by a twodimensional space, the optimal maintenance policy was presented by a two-dimensional policy map, shown in Fig. 10.

    Optimal maintenance actions on each decision epoch were shown in the policy map,and decision epochs were represented by aero-engine states. In the maintenance policy map of Fig. 10, the LLP state was regarded as the ordinate, and the performance state was regarded as the abscissa. Different maintenance actions were presented in different colors and shapes. In the legend, A0denoted action { Arep,0,Arec,0} ; A1denoted action { Arep,0,Arec,1} ; A2denoted action{ Arep,0,Arec,2} ;A4denoted action{ Arep,1,Arec,0} ;A6denoted action { Arep,1,Arec,2} . In engineering, a maintenance policy could be obtained according to the aero-engine state.

    Fig. 10 Maintenance policy map of the numerical experiment.

    4.5. Algorithm performance analysis

    In this section,a more complex numerical experiment was conducted to illustrate the proposed optimization algorithm with more detailed instructions. In engineering, an aero-engine is usually composed of more than one LLP type. Some LLP states were measured by flight-hour, different from the one in the first numerical experiment. Thus, a more complex numerical experiment with two LLP types was conducted.To distinguish the two numerical experiments,the first numerical experiment was named as Experiment 1,and the numerical experiment in this section was named as Experiment 2.

    Table 3 Transition matrices of no LLP replacement action.

    Table 4 Transition probability matrices of LLP1 replacement actions.

    4.5.1. Aero-engine states

    Two LLP types were considered in Experiment 2,and the two LLP states were measured by different units: one was flightcycle and the other was flight-hour.Thus,the aero-engine state was denoted as a three-dimensional state space. The three dimensions included the performance state,the LLP state represented by flight-cycle(LLP1), and the LLP state represented by flight-hour (LLP2).

    Corresponding to Experiment 1, in addition to all-new states T0and L0, the LLP1 state was divided into two levels,denoted as {T0,T1,T2};the LLP2 state was divided into three levels, denoted as {L0,L1,L2,L3}. Same as in Experiment 1,the performance state was represented by five levels, and the random failure state was regarded as a specific ‘‘performance state”. Thus, the aero-engine state space was represented by

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state; Tjdenoted the LLP1 state;Lkdenoted the LLP2 state.

    4.5.2. Maintenance actions and state transition

    In Experiment 2, LLP1 and LLP2 replacement actions were denoted as ArepT={ArepT,0,ArepT,1} and ArepL={ArepL,0,ArepL,1}. Performance recovery actions were the same as those in Experiment 1. Performance recovery assumptions were also available.Maintenance actions in Experiment 2 were represented by

    Similar to LLP1, the LLP2 state would transfer as the flight-hour increasing, and transition probabilities were unavailable for LLP2. However, aero-engine state transition matrices were changed by LLP2.

    On Em, action Am={ Arec,i,ArepT,0,ArepL,0|i=0,1,2,3}denoted no LLP replaced, and transition matrices are presented in Table 3, denoted as p(S,Arec,i,ArepT,0,ArepL,0|i=0,1,2,3)= p3{PF,i,,PN,PR,i|i=0,1,2,3}. The concrete forms of PF,, and PRwere the same as those in Experiment 1.

    State transition matrices of Am={ Arec,i,ArepT,1,ArepL,0|i=0,1,2,3} are presented in Table 4, denoted as p(S,Arec,i,ArepT,1,ArepL,0|i = 0,1,2,3) = p4{ PF,i,,PN,PR,i|i = 0,1,2,3} .

    State transition matrices of Am={Arec,i,ArepT,0,ArepL,1|i=0,1,2,3} are presented in Table 5, denoted as p(S,Arec,i,ArepT,0,ArepL,1|i = 0,1,2,3) = p5{PF,i,,PN,PR,i|i = 0,1,2,3}.

    State transition matrices of Am={Arec,i,ArepT,1,ArepL,1|i=0,1,2,3} are presented in Table 6, denoted as p(S,Arec,i,ArepT,1,ArepL,1|i = 0,1,2,3) = p6{PF,i,,PN,PR,i|i = 0,1,2,3}.

    4.5.3. Maintenance policy optimization

    In Experiment 2, the reinforcement learning algorithm of Gauss-Seidel value iteration was also adopted to optimize the maintenance policy. Hypothetical costs were adopted,and the LLP1 replacement cost Cllp,1was assumed to be 300 while the LLP2 replacement cost Cllp,2was assumed to be 600. The three performance recovery action costs were assumed to be 200, 500, and 800, respectively. The discount factor was set as γ=0.9. Because the aero-engine state was represented by a three-dimensional space, including the LLP1, LLP2, and performance states, the optimal maintenance policy was represented by a three-dimensional policy map, shown in Fig. 11.

    Table 6 Transition probability matrices of LLP1 and LLP2 replacement actions.

    Fig. 11 Maintenance policy map of Experiment 2.

    Table 7 Algorithm information of two experiments.

    In the three-dimensional policy map,the LLP1,LLP2,and performance states were regarded as x axis, z axis, and y axis,respectively.The maintenance actions were presented in different colors and shapes.In the legend of the maintenance policy map in Fig. 11, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action{Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action {Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.Based on the policy map, a maintenance policy was obtained according to the aero-engine state.

    4.5.4. Algorithm analysis

    In aforementioned two numerical experiments, a twodimensional state space was adopted in Experiment 1, while a three-dimensional state space was adopted in Experiment 2.It was obvious that state transition matrices of Experiment 2 were more complicated. Thus, the algorithm would become complex as the state number increased. The algorithm information of the two numerical experiments is shown in Table 7.

    In the aforementioned numerical experiments,performance and random failure states were defined as probability states,because they transferred by probabilities. LLP states did not transfer by probabilities, and were defined as definite states.As shown in the two numerical experiments, state transition matrix forms were impacted by definite states. Transition probabilities were impacted by probability states.In reinforcement learning, a larger state space would make the algorithm more complicated, and more iterations were needed to seek for the optimal policy. The impact of aero-engine state space complexity on the algorithm was obvious.

    As the discount factor is an important coefficient in reinforcement learning, the discount factor impact was analyzed by contrast experiments, which were based on Experiment 1.In addition to the discount factor, the other parameters were all the same as those in Experiment 1. Maintenance policy maps of discount factor analysis are shown in subgraphs of Fig. 12.

    Fig. 12 Policy maps of discount factor analysis in Example 1.

    In the policy maps,different maintenance actions were represented by different colors and shapes. In the legend of Fig. 12, A0denoted action {Arep,0,Arec,0}; A1denoted action{Arep,0,Arec,1}; A2denoted action {Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A6denoted action {Arep,1,Arec,2}. The analysis experiments showed that when the discount factor was set as γ >0.43,the optimal policy maps were the same as those in Fig. 10;when the discount factor was set as γ <0.35, the optimal policy maps were the same as those in Fig. 12(c). In Fig. 12, discount factors were set as γ = 0.43, γ = 0.41, and γ = 0.35,respectively. The analysis experiments showed that when a smaller discount factor was adopted, more low-cost maintenance actions were adopted in the optimal maintenance policy.It was consistent with the aforementioned analysis of the discount factor.

    As hypothetical maintenance costs were adopted in numerical experiments, cost impacts were analyzed. Based on the aforementioned experiments, the medium performance recovery cost was set as Crep,2=350, and it was regarded as the benchmark cost. Maintenance policy maps of different cost ratios are shown in subgraphs of Figs. 13 and 14.

    Fig. 13 Policy maps of performance recovery cost analysis in Example 1.

    In the legend of Fig. 13, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A3denoted action {Arep,0,Arec,3}; A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A7denoted action {Arep,1,Arec,3}. In the legend of Fig. 14, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A3denoted action {Arec,3,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A8denoted action {Arec,3,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1};A14denoted action{Arec,3,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.

    Fig. 14 Policy maps of performance recovery cost analysis in Example 2.

    Fig. 15 Policy maps of LLP replacement cost analysis in Example 1.

    As shown in Figs.13(a)and 14(a),showed the optimal policy when Crec,1decreased; Figs. 13(b) and 14(b) showed the optimal policy when Crec,3decreased; Figs. 13(c) and 14(c)showed the optimal policy when Crec,1and Crec,3decreased simultaneously. As shown in Figs. 13(c) and 14(c), when Crec,1and Crec,3decreased simultaneously, optimal policy changes were not obvious.

    The impact of the LLP replacement cost on the optimal maintenance policy was analyzed by contrast experiments.Experiment results showed that the optimal maintenance policy would not change as the LLP replacement cost increasing.However,LLP replacement action times would increase as the LLP replacement cost decreasing.Optimal policy maps of LLP replacement cost analysis are shown in Figs. 15-17.

    In the legend of Fig. 15, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A6denoted action {Arep,1,Arec,2}. In the legends of Figs. 16 and 17, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A2denoted action {Arec,2,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A7denoted action {Arec,2,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A13denoted action {Arec,2,ArepT,1,ArepL,1}; A15denoted action{Arec,0,ArepT,1,ArepL,1}.

    As shown in Fig. 15, the LLP replacement cost decreased from subgraph (a) to (c). In Figs. 16 and 17, LLP1 and LLP2 replacement costs decreased from subgraph (a) to (b).It was shown that LLP replacement times would increase as the LLP replacement cost decreasing, and policy changes appeared on interface decision epochs.

    In the aforementioned experiments, the impact of the LLP residual life was not considered. Thus, based on the assumption that the random failure probability would increase as the LLP residual life decreasing, contrast experiments were performed to analyze the optimization algorithm.Optimal policy maps are shown in Fig. 18.

    Fig. 16 Policy maps of LLP1 replacement cost analysis in Example 2.

    Fig. 17 Policy maps of LLP2 replacement cost analysis in Example 2.

    In the legend of Fig. 18, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}. Based on Experiment 2, according to the residual life of the elder LLP, random failure probabilities were enhanced by 5%, 10%, and 15% respectively from subgraph (a) to (c)of Fig. 18. Policy maps showed no variation. Thus, the LLP residual life may not affect the optimal maintenance policy.

    5. Conclusions

    Based on the reinforcement learning approach,an aero-engine life-cycle maintenance policy optimization algorithm was proposed, which was able to address the long service-life and the hybrid maintenance strategy synchronously. To address the hybrid maintenance strategy,the multi-dimensional state space was adopted to represent the aero-engine state. Based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the life-cycle maintenance policy.

    Compared with traditional optimization methods, the optimal maintenance policy was used to indicate when and how to repair an aero-engine, taking the place of maintenance intervals and work-scopes in traditional methods.Meanwhile, the aero-engine long service-life, the hybrid maintenance strategy, and random factor destabilization were all addressed by the proposed optimization algorithm.Because few historical data was available for training the pre-specified optimization model of the aero-engine lifecycle maintenance policy, the reinforcement learning approach provided an appropriate way. In the reinforcement learning framework, the aero-engine state space, maintenance actions, and state transition matrices were determined according to aero-engine real-life operation. The Gauss-Seidel value iteration algorithm was employed to solve the long-term decision-making problem. The proposed optimization algorithm would help in making a wiser aero-engine life-cycle maintenance policy, resulting in a lower life-cycle maintenance cost. Two numerical experiments and algorithm analyses were employed to illustrate the proposed optimization algorithm in detail.

    As real aero-engine maintenance cost data was unavailable,hypothetical data was adopted in the numerical experiments.In future studies, maintenance cost calculation methods deserve further attention to improve the applicability of the proposed optimization algorithm.

    Fig. 18 Policy maps with LLP lifetime impact on transition probability.

    Acknowledgments

    The authors thank anonymous reviewers for their critical and constructive review of the manuscript. This work was cosupported by the Key National Natural Science Foundation of China (No. U1533202), the Civil Aviation Administration of China (No. MHRD20150104), and the Shandong Independent Innovation and Achievements Transformation Fund,China (No. 2014CGZH1101).

    欧美日韩精品成人综合77777| 成人国产麻豆网| 身体一侧抽搐| 欧美日韩精品成人综合77777| 波多野结衣高清作品| 极品教师在线免费播放| 欧美色视频一区免费| 久99久视频精品免费| 成年女人毛片免费观看观看9| 国产色婷婷99| 亚洲电影在线观看av| 日本与韩国留学比较| 天堂动漫精品| 给我免费播放毛片高清在线观看| 真人一进一出gif抽搐免费| 一区二区三区免费毛片| 很黄的视频免费| 色视频www国产| 亚洲成人久久爱视频| av福利片在线观看| 国产在视频线在精品| 日本三级黄在线观看| 欧美国产日韩亚洲一区| 99久久成人亚洲精品观看| 国产av麻豆久久久久久久| 欧美激情久久久久久爽电影| 欧美人与善性xxx| 国产探花极品一区二区| 国产精品人妻久久久久久| 麻豆一二三区av精品| 亚洲成a人片在线一区二区| 国产真实乱freesex| 狂野欧美白嫩少妇大欣赏| 国产成人av教育| 乱系列少妇在线播放| 成人性生交大片免费视频hd| 在线免费十八禁| 能在线免费观看的黄片| 亚洲人与动物交配视频| 在线看三级毛片| xxxwww97欧美| 亚洲精华国产精华液的使用体验 | 天堂影院成人在线观看| 久久国产精品人妻蜜桃| 日韩精品中文字幕看吧| 国产精品无大码| 国产成人福利小说| 嫩草影院新地址| 亚洲va在线va天堂va国产| 给我免费播放毛片高清在线观看| 老司机福利观看| 日韩在线高清观看一区二区三区 | 亚洲精品影视一区二区三区av| 麻豆av噜噜一区二区三区| 亚洲av中文字字幕乱码综合| 国产乱人视频| 国产精品人妻久久久影院| 一区二区三区四区激情视频 | 亚洲精品影视一区二区三区av| 桃色一区二区三区在线观看| 看片在线看免费视频| 亚洲中文字幕日韩| 色综合亚洲欧美另类图片| 男女之事视频高清在线观看| 亚洲精品456在线播放app | 女生性感内裤真人,穿戴方法视频| 亚洲精品久久国产高清桃花| 老女人水多毛片| 亚洲专区国产一区二区| 色吧在线观看| 免费观看的影片在线观看| 亚洲精品乱码久久久v下载方式| 天天一区二区日本电影三级| 精品久久久久久久久亚洲 | 欧美不卡视频在线免费观看| 亚洲精品日韩av片在线观看| 91久久精品国产一区二区成人| 99热这里只有是精品在线观看| 成人欧美大片| 日韩人妻高清精品专区| 亚洲精品粉嫩美女一区| 在线免费观看不下载黄p国产 | 国产一区二区亚洲精品在线观看| 美女cb高潮喷水在线观看| 中文字幕免费在线视频6| 日韩国内少妇激情av| av在线观看视频网站免费| 成年女人永久免费观看视频| 1024手机看黄色片| 国产又黄又爽又无遮挡在线| 欧美日韩综合久久久久久 | 很黄的视频免费| 国产欧美日韩精品一区二区| 欧美+亚洲+日韩+国产| 国产成人福利小说| 很黄的视频免费| 成人欧美大片| 亚洲 国产 在线| 久久久久久久久大av| 亚洲av成人av| 老熟妇仑乱视频hdxx| 精品久久久久久成人av| 91精品国产九色| 在线看三级毛片| 桃色一区二区三区在线观看| 在线免费观看不下载黄p国产 | 又黄又爽又刺激的免费视频.| 日韩欧美在线二视频| 又爽又黄a免费视频| 99热这里只有是精品50| 日日摸夜夜添夜夜添小说| 国产亚洲精品av在线| 国产精品乱码一区二三区的特点| 在线观看免费视频日本深夜| 国产高潮美女av| 亚洲精品亚洲一区二区| 日本色播在线视频| 男女做爰动态图高潮gif福利片| a级毛片免费高清观看在线播放| 国产精品自产拍在线观看55亚洲| 一区二区三区四区激情视频 | 亚洲熟妇中文字幕五十中出| 精品乱码久久久久久99久播| 变态另类丝袜制服| 国产爱豆传媒在线观看| 夜夜看夜夜爽夜夜摸| 日本黄色视频三级网站网址| 久久热精品热| www日本黄色视频网| 97超视频在线观看视频| 狂野欧美激情性xxxx在线观看| 精品免费久久久久久久清纯| a在线观看视频网站| 麻豆精品久久久久久蜜桃| 亚洲久久久久久中文字幕| 日韩大尺度精品在线看网址| 亚洲精华国产精华液的使用体验 | 久久午夜福利片| 国产麻豆成人av免费视频| 欧洲精品卡2卡3卡4卡5卡区| 国产精品亚洲一级av第二区| 97超视频在线观看视频| 免费搜索国产男女视频| 国产精品久久电影中文字幕| 成年版毛片免费区| АⅤ资源中文在线天堂| 18禁黄网站禁片午夜丰满| 国产激情偷乱视频一区二区| 日本与韩国留学比较| 欧美激情在线99| 九九爱精品视频在线观看| 欧美日韩综合久久久久久 | 国产精品永久免费网站| 日本黄大片高清| 少妇的逼好多水| 国产真实乱freesex| 亚洲中文日韩欧美视频| 久久亚洲精品不卡| 亚洲va在线va天堂va国产| 啦啦啦观看免费观看视频高清| 男人舔奶头视频| 欧美一区二区亚洲| 精品99又大又爽又粗少妇毛片 | 久久午夜亚洲精品久久| 亚洲成av人片在线播放无| 少妇人妻一区二区三区视频| www.www免费av| 日韩大尺度精品在线看网址| 嫩草影视91久久| 国产成年人精品一区二区| 亚洲无线在线观看| 99久久无色码亚洲精品果冻| 国产亚洲精品久久久com| 99热只有精品国产| 国产成人福利小说| 97热精品久久久久久| 亚洲人与动物交配视频| 久久精品国产鲁丝片午夜精品 | 精品久久久久久久久亚洲 | 精品一区二区三区av网在线观看| 国产精品亚洲美女久久久| 亚洲电影在线观看av| av国产免费在线观看| 男女边吃奶边做爰视频| 国模一区二区三区四区视频| 国产v大片淫在线免费观看| 人妻丰满熟妇av一区二区三区| 少妇熟女aⅴ在线视频| 12—13女人毛片做爰片一| 丰满的人妻完整版| x7x7x7水蜜桃| 日本-黄色视频高清免费观看| 床上黄色一级片| aaaaa片日本免费| 少妇高潮的动态图| 久久久久久大精品| 久久久久久久久久久丰满 | 制服丝袜大香蕉在线| 亚洲图色成人| 波多野结衣高清无吗| 91在线精品国自产拍蜜月| 国产精品永久免费网站| 亚洲美女黄片视频| 久久久色成人| 成年版毛片免费区| 中文字幕熟女人妻在线| 女的被弄到高潮叫床怎么办 | 精品欧美国产一区二区三| 色综合婷婷激情| 搡老岳熟女国产| 日本 欧美在线| 啦啦啦韩国在线观看视频| 亚洲av中文av极速乱 | 动漫黄色视频在线观看| 我要搜黄色片| 免费人成在线观看视频色| 99国产极品粉嫩在线观看| 久久久久久久精品吃奶| av专区在线播放| 成人三级黄色视频| 日本一二三区视频观看| 午夜日韩欧美国产| 国产私拍福利视频在线观看| xxxwww97欧美| 国产一区二区三区视频了| 中文字幕熟女人妻在线| 国产大屁股一区二区在线视频| 久久久久久九九精品二区国产| 校园人妻丝袜中文字幕| 亚洲在线观看片| 亚洲精品一区av在线观看| 天天躁日日操中文字幕| 中文资源天堂在线| 国产私拍福利视频在线观看| 性欧美人与动物交配| 国产高清不卡午夜福利| 极品教师在线免费播放| 啦啦啦观看免费观看视频高清| 男人舔奶头视频| a级毛片a级免费在线| 亚洲精品一区av在线观看| 97碰自拍视频| 国产亚洲91精品色在线| 美女cb高潮喷水在线观看| 在线播放国产精品三级| 亚洲色图av天堂| 日韩欧美国产一区二区入口| 色综合亚洲欧美另类图片| 精品久久久久久久久av| 在线a可以看的网站| 日本五十路高清| 哪里可以看免费的av片| 97热精品久久久久久| 黄色丝袜av网址大全| 欧美区成人在线视频| 欧美黑人欧美精品刺激| 成人性生交大片免费视频hd| 国产在线男女| 成人毛片a级毛片在线播放| 亚洲av成人av| 日本欧美国产在线视频| 国产在线精品亚洲第一网站| 国产熟女欧美一区二区| 免费电影在线观看免费观看| 亚洲va在线va天堂va国产| 乱码一卡2卡4卡精品| 熟女电影av网| 国产黄a三级三级三级人| 一边摸一边抽搐一进一小说| 亚洲精品乱码久久久v下载方式| 亚洲最大成人中文| 久久久久久久午夜电影| 级片在线观看| 欧美日韩精品成人综合77777| 好男人在线观看高清免费视频| 亚洲性久久影院| 国产91精品成人一区二区三区| 精品欧美国产一区二区三| 国产伦一二天堂av在线观看| 国产欧美日韩精品亚洲av| 桃色一区二区三区在线观看| 麻豆成人av在线观看| or卡值多少钱| 成人美女网站在线观看视频| 国内揄拍国产精品人妻在线| 夜夜夜夜夜久久久久| 日本黄色视频三级网站网址| 亚洲国产精品成人综合色| 大又大粗又爽又黄少妇毛片口| 国产真实伦视频高清在线观看 | 亚洲色图av天堂| 琪琪午夜伦伦电影理论片6080| 亚洲美女搞黄在线观看 | 少妇的逼好多水| 亚洲精品456在线播放app | 少妇裸体淫交视频免费看高清| 日韩欧美在线二视频| 亚洲中文字幕日韩| 欧美一区二区亚洲| 联通29元200g的流量卡| 一进一出好大好爽视频| 99热这里只有精品一区| av国产免费在线观看| 国产真实乱freesex| 免费无遮挡裸体视频| 久久久久久久午夜电影| 99热这里只有是精品50| 国产精品国产三级国产av玫瑰| 久久久国产成人精品二区| 动漫黄色视频在线观看| 69人妻影院| 精品久久久噜噜| 黄片wwwwww| 熟女人妻精品中文字幕| 我的老师免费观看完整版| 午夜老司机福利剧场| 中国美白少妇内射xxxbb| 美女高潮喷水抽搐中文字幕| 女的被弄到高潮叫床怎么办 | 国产成人aa在线观看| 日韩,欧美,国产一区二区三区 | 亚洲成av人片在线播放无| 日本精品一区二区三区蜜桃| 亚洲精品乱码久久久v下载方式| 日本与韩国留学比较| 国产人妻一区二区三区在| 白带黄色成豆腐渣| 欧美一区二区亚洲| 国产精品不卡视频一区二区| 男女边吃奶边做爰视频| 精品久久久噜噜| 久久久成人免费电影| 在线看三级毛片| 国产精品一区二区性色av| 亚洲黑人精品在线| 亚洲,欧美,日韩| 色哟哟·www| 日本一本二区三区精品| 一进一出好大好爽视频| 国产精品久久久久久久久免| 人人妻,人人澡人人爽秒播| 禁无遮挡网站| 国产精品亚洲一级av第二区| 嫩草影院入口| 国产精品一区二区三区四区免费观看 | 精品人妻视频免费看| 久99久视频精品免费| 日韩中文字幕欧美一区二区| a在线观看视频网站| 老熟妇乱子伦视频在线观看| 99九九线精品视频在线观看视频| 精品99又大又爽又粗少妇毛片 | 日韩欧美免费精品| 乱系列少妇在线播放| 天堂影院成人在线观看| 国产精品国产高清国产av| 亚洲专区中文字幕在线| 精品免费久久久久久久清纯| 蜜桃久久精品国产亚洲av| 九九在线视频观看精品| 长腿黑丝高跟| 欧美中文日本在线观看视频| 亚洲专区国产一区二区| 99在线人妻在线中文字幕| 麻豆av噜噜一区二区三区| 在线观看66精品国产| 有码 亚洲区| 成人二区视频| 一级黄色大片毛片| 亚洲av美国av| x7x7x7水蜜桃| 午夜免费男女啪啪视频观看 | 男女啪啪激烈高潮av片| 99国产精品一区二区蜜桃av| 九色国产91popny在线| 国产一区二区三区av在线 | 久久久久久久久大av| 变态另类丝袜制服| 99精品在免费线老司机午夜| 床上黄色一级片| 国产色婷婷99| 久久久久久久午夜电影| 午夜日韩欧美国产| 久久精品国产自在天天线| 成人鲁丝片一二三区免费| 最好的美女福利视频网| 97超级碰碰碰精品色视频在线观看| 国内少妇人妻偷人精品xxx网站| 免费观看的影片在线观看| 日韩国内少妇激情av| 中文字幕熟女人妻在线| 成人美女网站在线观看视频| 99在线人妻在线中文字幕| 精品国内亚洲2022精品成人| 精品久久国产蜜桃| 精品久久久久久久人妻蜜臀av| 国产男人的电影天堂91| 精品久久久久久成人av| 日本色播在线视频| 欧洲精品卡2卡3卡4卡5卡区| 少妇丰满av| 亚洲不卡免费看| 久久亚洲精品不卡| 国产精品久久久久久久久免| 欧美日韩精品成人综合77777| 99热这里只有是精品在线观看| 22中文网久久字幕| 很黄的视频免费| 婷婷精品国产亚洲av在线| 久久久色成人| 亚洲专区中文字幕在线| 最新在线观看一区二区三区| 国产一区二区三区av在线 | 麻豆国产av国片精品| 麻豆久久精品国产亚洲av| 久9热在线精品视频| 我的老师免费观看完整版| 熟女人妻精品中文字幕| 日本五十路高清| 成年免费大片在线观看| 午夜a级毛片| 日本在线视频免费播放| 亚洲av.av天堂| 老师上课跳d突然被开到最大视频| 久久久久久久久久黄片| 日韩大尺度精品在线看网址| 久久精品综合一区二区三区| 亚洲无线观看免费| 中文字幕熟女人妻在线| 两性午夜刺激爽爽歪歪视频在线观看| 91久久精品国产一区二区成人| 国产黄片美女视频| 久久久久国内视频| 国产一级毛片七仙女欲春2| 成年人黄色毛片网站| 人妻夜夜爽99麻豆av| 午夜爱爱视频在线播放| 亚洲美女黄片视频| 精品久久久久久成人av| 高清日韩中文字幕在线| 成人美女网站在线观看视频| 色综合站精品国产| 成人永久免费在线观看视频| 毛片一级片免费看久久久久 | 少妇裸体淫交视频免费看高清| 两性午夜刺激爽爽歪歪视频在线观看| 亚洲欧美日韩卡通动漫| 在线观看免费视频日本深夜| 一个人看视频在线观看www免费| 亚洲国产欧洲综合997久久,| 国产在线精品亚洲第一网站| 欧美+亚洲+日韩+国产| 高清毛片免费观看视频网站| 身体一侧抽搐| 日韩 亚洲 欧美在线| 国内少妇人妻偷人精品xxx网站| 可以在线观看的亚洲视频| 久久热精品热| xxxwww97欧美| 五月伊人婷婷丁香| 亚洲不卡免费看| 美女黄网站色视频| 国产探花在线观看一区二区| 舔av片在线| 成人永久免费在线观看视频| 毛片一级片免费看久久久久 | av在线天堂中文字幕| 两性午夜刺激爽爽歪歪视频在线观看| 亚洲aⅴ乱码一区二区在线播放| 白带黄色成豆腐渣| 在线观看午夜福利视频| 国产一级毛片七仙女欲春2| 精品一区二区三区视频在线| 久久久久久九九精品二区国产| 热99re8久久精品国产| 亚洲精品亚洲一区二区| 麻豆精品久久久久久蜜桃| 精品久久久久久,| 如何舔出高潮| 久久精品夜夜夜夜夜久久蜜豆| 国产又黄又爽又无遮挡在线| 久久人人爽人人爽人人片va| 在线观看舔阴道视频| 校园人妻丝袜中文字幕| 亚洲人成网站在线播放欧美日韩| 真人一进一出gif抽搐免费| 午夜福利高清视频| 男人狂女人下面高潮的视频| 亚洲综合色惰| 联通29元200g的流量卡| 制服丝袜大香蕉在线| 色综合色国产| 毛片女人毛片| 欧美又色又爽又黄视频| 欧美另类亚洲清纯唯美| 久久久久久久久久黄片| 韩国av一区二区三区四区| 成人午夜高清在线视频| 在线a可以看的网站| 高清毛片免费观看视频网站| 91午夜精品亚洲一区二区三区 | 我的女老师完整版在线观看| 啦啦啦观看免费观看视频高清| 极品教师在线免费播放| 婷婷精品国产亚洲av| 99久久精品国产国产毛片| 又粗又爽又猛毛片免费看| 极品教师在线视频| 美女黄网站色视频| 国产亚洲精品av在线| 嫩草影院入口| 日韩一本色道免费dvd| 国产69精品久久久久777片| 亚洲综合色惰| 中文字幕高清在线视频| 亚洲aⅴ乱码一区二区在线播放| av.在线天堂| 国产视频内射| 他把我摸到了高潮在线观看| 午夜免费男女啪啪视频观看 | 欧美极品一区二区三区四区| 国产精品一区二区三区四区久久| 成人永久免费在线观看视频| 亚洲aⅴ乱码一区二区在线播放| 精品福利观看| 夜夜爽天天搞| 久久精品国产亚洲av天美| 直男gayav资源| 亚洲一级一片aⅴ在线观看| 日韩欧美国产在线观看| 十八禁网站免费在线| 成人亚洲精品av一区二区| 免费看a级黄色片| 欧美日韩乱码在线| 国产探花极品一区二区| 在线观看美女被高潮喷水网站| 国内精品一区二区在线观看| 毛片一级片免费看久久久久 | 全区人妻精品视频| 免费搜索国产男女视频| 三级男女做爰猛烈吃奶摸视频| 欧美性猛交黑人性爽| 天堂av国产一区二区熟女人妻| 亚洲自偷自拍三级| 国产av一区在线观看免费| 深夜精品福利| 联通29元200g的流量卡| 成人特级av手机在线观看| 99热这里只有精品一区| www.色视频.com| 99久久成人亚洲精品观看| 欧美黑人欧美精品刺激| 国产av一区在线观看免费| 久久精品久久久久久噜噜老黄 | 国产精品伦人一区二区| 国产老妇女一区| 日本 欧美在线| 999久久久精品免费观看国产| 国产精品久久视频播放| av福利片在线观看| 午夜老司机福利剧场| 999久久久精品免费观看国产| 国产成人av教育| 精品久久久久久,| 亚洲av电影不卡..在线观看| 搡女人真爽免费视频火全软件 | 别揉我奶头~嗯~啊~动态视频| 一区二区三区高清视频在线| 国产成人av教育| 91久久精品电影网| 在线免费观看不下载黄p国产 | 99热只有精品国产| 日本一本二区三区精品| 哪里可以看免费的av片| 久久人妻av系列| 国产欧美日韩精品亚洲av| 看片在线看免费视频| 亚洲精品国产成人久久av| netflix在线观看网站| 最近视频中文字幕2019在线8| 一个人看视频在线观看www免费| 免费人成视频x8x8入口观看| 在线观看免费视频日本深夜| 成年人黄色毛片网站| 91午夜精品亚洲一区二区三区 | 亚洲乱码一区二区免费版| 特大巨黑吊av在线直播| www.色视频.com| 国内精品久久久久精免费| a级毛片a级免费在线| 免费看光身美女| 97热精品久久久久久| 国产高潮美女av| 99热这里只有是精品在线观看| 亚洲欧美日韩无卡精品| 成熟少妇高潮喷水视频| 最新中文字幕久久久久| 在线免费观看不下载黄p国产 | 亚洲精品影视一区二区三区av| 亚洲中文字幕一区二区三区有码在线看| 一进一出抽搐动态| 亚洲精品日韩av片在线观看| 国产在视频线在精品| 床上黄色一级片| 亚洲,欧美,日韩| 一区二区三区免费毛片| 老熟妇仑乱视频hdxx| 一级av片app| www.色视频.com| 成人欧美大片| 亚洲av第一区精品v没综合| 在线观看66精品国产| 可以在线观看的亚洲视频| 精华霜和精华液先用哪个| 男女那种视频在线观看|