• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach

    2019-10-26 01:12:16ZhenLIShishengZHONGLinLIN
    CHINESE JOURNAL OF AERONAUTICS 2019年9期

    Zhen LI, Shisheng ZHONG, Lin LIN

    School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China

    KEYWORDS Aero-engine;Hybrid strategy;Maintenance policy;Optimization algorithm;Reinforcement learning

    Abstract An aero-engine maintenance policy plays a crucial role in reasonably reducing maintenance cost. An aero-engine is a type of complex equipment with long service-life. In engineering,a hybrid maintenance strategy is adopted to improve the aero-engine operational reliability. Thus,the long service-life and the hybrid maintenance strategy should be considered synchronously in aero-engine maintenance policy optimization. This paper proposes an aero-engine life-cycle maintenance policy optimization algorithm that synchronously considers the long service-life and the hybrid maintenance strategy. The reinforcement learning approach was adopted to illustrate the optimization framework, in which maintenance policy optimization was formulated as a Markov decision process. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the maintenance policy. Compared with traditional aero-engine maintenance policy optimization methods,the long service-life and the hybrid maintenance strategy could be addressed synchronously by the proposed algorithm. Two numerical experiments and algorithm analyses were performed to illustrate the optimization algorithm in detail.

    1. Introduction

    An aero-engine, which is composed of mechanic-electrichydraulic coupling systems,is the power plant of an aircraft.1,2It has been reported that more than 30% of aircraft mechanical problems are related to aero-engines, and the aero-engine maintenance cost contributes to about 30% of an airline's direct operating cost.3A maintenance optimization method provides an available way to reduce the maintenance cost reasonably.4In general, excessive maintenance is costly, while insufficient maintenance may lead to disasters.Thus,a maintenance policy plays a crucial role in balancing the maintenance cost and operational reliability.5However, it is not an easy work to optimize an aero-engine maintenance policy manually,especially taking the long service-life and the hybrid maintenance strategy into consideration synchronously.

    In engineering,a hybrid maintenance strategy is adopted to improve the civil aero-engine operational reliability.Strategies of Condition-Based Maintenance (CBM), Hard-time Maintenance (HM), and failure Corrective Maintenance (CM) are included in the hybrid maintenance strategy.6As aero-engine performance deterioration is inevitable,7gas path performance parameters are monitored for CBM. The Life Limit Part(LLP) should be replaced before its life limitation,8and HM is adopted. Moreover, CM is performed when an aeroengine is in the random failure state.Thus,the hybrid maintenance strategy should be considered in aero-engine maintenance policy optimization. However, few existing aero-engine maintenance optimization methods are able to address the hybrid strategy. At the same time, an aero-engine is a type of equipment with long service-life,9which should be synchronously considered in maintenance policy optimization.To the best of our knowledge, this is the first paper to investigate an aero-engine maintenance policy optimization method that can synchronously address the hybrid maintenance strategy and long service-life.

    In traditional aero-engine maintenance optimization methods, maintenance intervals are used to decide when to repair an aero-engine, and maintenance work-scopes indicate how to carry out maintenance actions. Maintenance intervals and work-scopes can be obtained by traditional separate models.For example, LLP replacement intervals and performance recovery intervals can be optimized separately by current models.10-12Based on maintenance intervals, maintenance workscopes are obtained by traditional decision-making models.13,14Based on traditional optimization methods, an aeroengine maintenance decision support system was proposed by Fu et al.,15in which maintenance interval and work-scope optimization models were presented. An optimization method for reliability-centered maintenance was proposed by Crocker and Kumar,16in which the concepts of soft life and hard life were used to optimize a military aero-engine maintenance policy.A multi-objective evolutionary algorithm was also adopted to solve the aero-engine maintenance scheduling problem,17taking the module exchange into consideration. To optimize an aero-engine maintenance policy, traditional optimization methods would become extremely complicated when the long service-life and the hybrid maintenance strategy are considered synchronously.18

    In general, machine learning methods include supervised learning, unsupervised learning, and reinforcement learning,19and the reinforcement learning method has attracted increasing interests in solving decision-making problems.20Reinforcement learning represents a machine learning method in which an agent learns how to behave through action rewards.Different from widely used supervised learning methods, there is no presentation of input and output pairs in reinforcement learning. In reinforcement learning, an agent chooses an available action according to the environment on each decision epoch.The chosen action changes the environment, along with a reward to the agent. The objective of the agent is to find the action collection, whose reward is maximal in the long run.

    Reinforcement learning methods have been adopted in energy system charging policy optimization,21,22energy system and distributed system schedule determination,23,24multiple robotic task optimization,25demand response optimization,26,27robust control optimization,28multiple satellites task planning,29et al. Although reinforcement learning methods have been successfully applied, they have not aroused much attention in aero-engine maintenance policy optimization. Reinforcement learning does provide a more appropriate way for aero-engine life-cycle maintenance policy optimization. Thus, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on reinforcement learning. The main contributions of this paper are as follows:

    (1) An aero-engine life-cycle maintenance policy optimization algorithm is proposed, which can synchronously address the aero-engine long service-life and the hybrid maintenance strategy.

    (2) To address the hybrid strategy, the aero-engine state is represented by a multi-dimensional state space.

    (3) Reinforcement learning is adopted to illustrate the maintenance policy optimization. In the reinforcement learning framework, maintenance policy optimization is formulated as a discrete Markov Decision Process(MDP), and the Gauss-Seidel value iteration algorithm is adopted to optimize the maintenance policy.

    The remainder of this paper is organized as follows.Section 2 introduces aero-engine maintenance policy optimization and the reinforcement learning approach, and the deficiencies of traditional optimization methods are analyzed.In Section 3, the proposed aero-engine life-cycle maintenance policy optimization algorithm is described in detail. In Section 4, two simulation experiments and algorithm analysis are used to illustrate the proposed optimization algorithm.Conclusions and future work are discussed in Section 5.

    2. Aero-engine maintenance policy optimization and reinforcement learning

    2.1. Aero-engine maintenance policy optimization

    A maintenance policy indicates when and how to repair an aero-engine. Traditionally, maintenance intervals indicate when to repair an aero-engine, which are obtained by traditional separate optimization models, addressing CBM, HM and,CM strategies separately.In traditional methods,maintenance work-scopes indicate how to repair an aero-engine,which are optimized based on the outputs of interval optimization models. In summary, there are several deficiencies in traditional aero-engine maintenance optimization methods, as follows:

    (1) In traditional methods,maintenance intervals and workscopes are optimized by separate models, and workscopes are optimized based on interval optimization results. Thus, the interactions of maintenance intervals and work-scopes are neglected.Moreover,interval optimization errors would be propagated to work-scope optimization.

    (2) Because the hybrid maintenance strategy is adopted for civil aero-engines, hybrid strategies should be addressed synchronously in optimization. However, traditional optimization methods address hybrid strategies separately, and interactions of hybrid strategies are neglected.

    (3) It is difficult for traditional optimization methods to address the aero-engine long-service life and the hybrid maintenance strategy synchronously.

    (4) Definite optimization results are obtained by traditional optimization methods. Due to random factors, definite optimization results may be poorly applicable in engineering.

    To deal with the aforementioned deficiencies of traditional methods, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on the reinforcement learning approach. Taking the place of maintenance intervals and work-scopes in traditional methods, the maintenance policy indicates when and how to repair an aero-engine.The proposed optimization algorithm is able to address the hybrid maintenance strategy and long service-life synchronously.

    To address the hybrid maintenance strategy, a multidimensional space is adopted to represent the aero-engine state.The maintenance strategies of CBM and CM are formulated as an MDP,and imperfect repair and random factors are all considered in state transition. Aero-engine maintenance policy optimization is illustrated by the reinforcement learning approach, and the Gauss-Seidel value iteration algorithm is adopted to seek for the optimal maintenance policy. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm makes it available to optimize the lifecycle maintenance policy. A comparison between traditional optimization methods and the proposed optimization algorithm is shown in Fig. 1.

    In Fig.1,traditional aero-engine maintenance optimization methods are shown on the left.Traditional methods are constituted by separate models,including an LLP interval optimization model, a performance interval optimization model, and a work-scope optimization model.LLP replacement and performance recovery intervals are obtained by separate optimization models. Based on the LLP replacement, performance recovery, and corrective maintenance intervals, the maintenance work-scope is optimized. The proposed aero-engine life-cycle maintenance policy optimization algorithm is shown on the right in Fig. 1. Based on the reinforcement learning framework, the strategies of HM, CM, and CBM are addressed synchronously by the proposed optimization algorithm. The traditional separate optimization models of the LLP replacement interval, the performance recovery interval,and the maintenance work-scope are replaced by the proposed optimization algorithm.

    2.2. Reinforcement learning approach

    Reinforcement learning is a machine learning method,which is widely used to solve multi-step, sequential-decision problems.Different from supervised learning, no pre-specified model is required in reinforcement learning. In aero-engine life-cycle maintenance policy optimization, few historical data is available for training the pre-specified mode. Thus, reinforcement learning provides a more appropriate way for aero-engine life-cycle maintenance policy optimization. Meanwhile, the aero-engine long service-life and the hybrid maintenance strategy can be addressed synchronously by reinforcement learning.

    In reinforcement learning, an agent takes on the work of optimizing an aero-engine maintenance policy. The agent is able to respond to dynamically changing aero-engine states through ongoing learning methods.30In aero-engine maintenance policy optimization, aero-engine states are represented by a multi-dimensional state space. To optimize the maintenance policy, the agent chooses a maintenance action according to the aero-engine state. The aero-engine state is changed by the chosen maintenance action,along with the maintenance cost, as shown in Fig. 2. The optimal objective of the agent is to find the maintenance action collection, whose total cost is the minimum in the long run.

    Fig. 2 Schematic diagram of reinforcement learning.

    Value iteration is a reinforcement learning algorithm that is widely adopted in solving decision-making problems. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm provides an appropriate way to optimize an aero-engine life-cycle maintenance policy. According to the Gauss-Seidel value iteration algorithm, an agent would run multiple episodes for the purpose of exploring and finding the optimal policy.The learning process is conducted for a sufficient number of iterations,and the total cost of each iteration is recorded.The minimum total cost is represented as Q-value,which is updated every iteration, and the Bellman equation is adopted as the updating mechanism in the Gauss-Seidel value iteration algorithm. The convergence of value iterative methods has been widely proven.31Thus, based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm.

    3. Aero-engine life-cycle maintenance policy optimization algorithm

    The reinforcement learning approach is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm, in which the hybrid maintenance strategy and long service-life are considered synchronously.In the reinforcement learning framework, the aero-engine state, maintenance actions, state transition matrices, and maintenance costs should be determined. To address the hybrid maintenance strategy, a multi-dimensional state space is adopted to represent the aero-engine state, taking performance, LLP, and random failure states into consideration synchronously. The optimal objective of the proposed optimization algorithm is to obtain a maintenance policy whose total cost is the minimum in the long run. The reinforcement learning algorithm of Gauss-Seidel value iteration is adopted, addressing longterm optimization. In this section, the proposed optimization algorithm is described in detail.sub-state Yjdenotes the performance state, and Zkdenotes the random failure state.

    (1) LLP state

    In engineering,an LLP must be replaced before its life limitation,and the HM strategy is adopted for LLP maintenance.Traditionally, the flight-cycle or flight-hour is adopted to represent the LLP state. For example, a fan shaft is an LLP of a CFM56-5B aero-engine, and its life limitation is 30,000 flightcycles. That is, a fan shaft must be replaced before 30,000 flight-cycles. For convenience, the LLP state is represented by discrete time increments.32In the proposed optimization algorithm, the LLP state is represented by several state levels,denoted as {Ti|i=0,1,2,...}, where T0denotes the all-new state; when m <n, state Tnis ‘‘older” than state Tm. It is defined that an LLP is in state Tn, when tn-1<tllp≤tn; tn-1and tnare boundary values; tllpis an LLP real state, measured by flight-cycle or flight-hour.

    (2) Performance state

    As the CBM strategy is adopted in aero-engine maintenance, the aero-engine performance state should also be considered in the proposed optimization algorithm. Most of the in-service aero-engines are equipped with condition monitoring systems, and performance parameters are sent to ground by the aircraft communication addressing and reporting system in quasi real-time. Traditionally, the aero-engine performance state is assessed by performance parameters.33,34In engineering, the Exhaust Gas Temperature Margin (EGTM)is adopted as an aero-engine performance indicator.35The EGTM is defined as the margin temperature between exhaust gas temperature and red-line temperature. The aero-engine performance degrades as it operates, presented as EGTM declining.36,37When the EGTM is close to the limitation, performance recovery maintenance should be performed. The 1000 flight-cycles EGTM parameters of a CFM56-5B aeroengine are shown in Fig. 3.

    As EGTM parameters are typical time series,38it would make the algorithm extremely complicated when EGTM time series are adopted directly.For convenience,the CBM strategy is formulated as a discrete MDP in the reinforcement learning framework. Thus, the aero-engine performance state is represented by several levels, denoted as {Di|i=0,1,2,...}, where D0denotes the all-new performance state; when m <n, state Dnis worse than state Dm. It is defined that the performance state is Dn, when dn-1<dper≤dn, in which dn-1and dnare

    3.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state is changed by performed actions. As the hybrid maintenance strategy is considered in the proposed optimization algorithm,a multi-dimensional state space is adopted to represent the aero-engine state, in which performance, LLP, and random failure states are considered synchronously. The multidimensional state space is represented by S={Xi,Yj,Zk,...},where Xi,Yj,Zk,... denote the sub-states in the multidimensional state space. Each sub-state denotes one considered factor, for example, sub-state Xidenotes the LLP state,the boundary values;dperis the EGTM real value,measured by centigrade. Thus, in the proposed optimization algorithm, the aero-engine performance state is indicated by the performance levels.

    Fig. 3 1000 flight-cycles EGTM of a CFM56-5B aero-engine.

    (3) Random failure state

    Although an aero-engine is with high reliability, it is subjected to random failures in practice. When an aero-engine is in the random failure state, CM should be performed to drive it back to the working state. Unlike the LLP or performance state, the random failure state is represented by two levels,denoted as {F0,F(xiàn)1}, where F0denotes the working state and F1denotes the random failure state.

    From the above,when LLP,performance,and random failure states are all considered, the aero-engine state is represented by a multi-dimensional state space, denoted as

    where Tidenotes the LLP state, Djdenotes the performance state, and Fkdenotes the random failure state.

    3.2. Maintenance actions

    In reinforcement learning, maintenance actions are performed on every decision epoch. The decision epoch is denoted as{Eii=| 0,1,2,...,m,...}, and state Siis regarded as the status between epochs Ei-1and Ei. As LLP, performance, and random failure states are all considered,LLP replacement actions,performance recovery actions, and failure corrective actions should be determined on each decision epoch.

    LLP replacement actions denote maintenance actions of replacing an aero-engine LLP. When an LLP replacement action is performed, the definite LLP is replaced, and the LLP state is changed.LLP replacement actions are represented by {Arep,i|i=0,1,2,...}, where Arep,0denotes no LLP replaced and Arep,m(m≠0) denotes LLP m replaced. When Arep,mis performed, the LLP m state is changed to the all-new state.

    Performance recovery actions denote maintenance actions of recovering aero-engine performance. When a performance recovery action is performed, the performance state is recovered by a definite level, and the performance state is changed.Performance recovery actions are represented by{Arec,j|j=0,1,2,...}, where Arec,0denotes no performance recovery action performed and Arec,m(m≠0)denotes the action of recovering m performance levels.

    Failure corrective actions denote maintenance actions of making a failure-state aero-engine back to the running state.When the aero-engine is trapped in the random failure state,a failure corrective action should be performed.Failure corrective actions are represented by Acor={Acor,0,Acor,1}, where Acor,0denotes no corrective maintenance performed and Acor,1denotes corrective maintenance performed.

    From the above, aero-engine maintenance actions are represented by

    Because an aero-engine does not operate during the maintenance process, maintenance actions are assumed to be ‘‘instantaneous” in the proposed optimization algorithm.39

    3.3. State transition

    In the reinforcement learning framework,the aero-engine state is changed by performed maintenance actions.Thus,LLP,performance, and random failure state transitions are illustrated as follows.

    (1) LLP state transition

    In engineering,the LLP state is measured by the flight-cycle or flight-hour,which increases directly as an aero-engine operates. The LLP state would be recovered to the all-new state when an LLP replacement action is performed. Thus, the LLP state transfers directly, without uncertainty.

    When action Arep,0is performed,LLP state Tiwould transfer to the definite state Ti+1,that is,p(Ti+1|Ti,Arep,0)=1.When action Arep,1is performed, LLP state Tiwould transfer to T0,that is, p(T0|Ti,Arep,1)=1. A schematic diagram of LLP state transition is shown in Fig. 4.

    (2) Performance state transition

    Aero-engine performance deterioration is inevitable in engineering. As the imperfect maintenance concept and random factors are considered in the proposed optimization algorithm,the CBM strategy is formulated as a discrete MDP. Probability matrices are adopted in performance state transition.

    (

    Because the maintenance concept of ‘‘a(chǎn)s good as new” has been proven to be far from the truth,40a more realistic concept of imperfect repair is adopted for performance recovery actions.41,42That is,the performance state cannot be recovered to the all-new state by any maintenance actions, and the performance state would transfer according to the transition probability matrices. When action Arec,m(m >0) is performed,the performance state would transfer from Dito Di-maccording to the probability matrix [p(Di-m|Di,Arec,m,i-m >0)]. A schematic diagram of performance state transition is shown in Fig. 5.

    The performance state transition probability matrices can be calculated by survival analysis based on the Weibull distribution.43,44

    Fig. 4 Schematic diagram of LLP state transition.

    Fig. 5 Schematic diagram of performance state transition.

    (3) Random failure state transition

    As an aero-engine may be in the random fault state occasionally, the CM strategy is also formulated as a discrete MDP. The random failure state would transfer according to probability matrices.

    In the proposed optimization algorithm, an aero-engine may fall into the random failure state F1by probability p(F1|Dj,F(xiàn)0;Acor,0). To be more realistic, it is assumed that the random failure probability is lower when the aero-engine is in a better performance state. Thus, the random failure probability p(F1|Dj,F(xiàn)0) is related to the performance state Dj.When the aero-engine is in the random failure state,corrective maintenance should be performed to drive it to the working state. It is assumed that the corrective maintenance is completely efficient, and the transition probability of the corrective maintenance is represented as p(F0|Dj,F(xiàn)1;Acor,1)=1.A schematic diagram of random failure state transition is shown in Fig. 6.

    From the above, the state transition matrix on Eiis represented by p(Si+1|Si,Ai), where Si+1denotes the aero-engine state on Ei+1, Sidenotes the aero-engine state on Ei, and Aidenotes the performed maintenance action on Ei. Different sub-states would transfer according to different modes, as illustrated above.

    3.4. Total cost and optimization

    In reinforcement learning, an agent chooses maintenance actions according to action costs.44,45In the proposed optimization algorithm, the optimal objective is to obtain a maintenance policy whose total cost is the minimum in the long run.The maintenance cost on decision epoch Ekis calculated by

    where ckdenotes the maintenance cost on decision epoch Ek;Cope,kdenotes the operating cost; Crep,k,Crec,k, Ccor,k, and Cotherdenote the LLP replacement cost, the performance recovery cost,the corrective maintenance cost,and other costs,respectively.

    Fig. 6 Schematic diagram of random failure state transition.

    In general, a good-performance-state aero-engine would have a better economic efficiency and a lower random failure rate. Thus, Cope,kis determined by the performance state.When an LLP replacement action is performed, material and replacement costs are both counted in Crep,k(Arep,k). LLP replacement costs vary from different LLP replacement actions. The performance recovery cost is represented by Crec,k(Arec,k), and when m >n, Crec,i(Arec,m)>Crec,i(Arec,n).The corrective maintenance cost Ccor,kis counted when a corrective maintenance is performed.

    As the life-cycle maintenance policy is optimized by the proposed optimization algorithm, the future maintenance cost should be counted in optimization. In reinforcement learning,a discount factor is adopted to address long-term optimization.Thus,the optimal objective of the proposed optimization algorithm is denoted as

    where C denotes the discounted future cost, and γ(γ ∈[0,1])denotes the discount factor, representing the relative impact of future action costs.

    In reinforcement learning, when a larger γ is adopted,future action costs would leave a greater impact on maintenance action selection.That is,when γ=0,the optimized policy is shortsighted,and the maintenance action is chosen by the current cost; when γ=1, all the future actions are considered in action selection,which would bring a heavy calculation burden. Thus, a balance between future costs and the calculation burden should be determined. Thus, the discount factor γ should be set as γ ∈(0,1), for example, γ=0.9.

    As the Gauss-Seidel value iteration algorithm is an effective reinforcement learning algorithm, which is widely used in policy optimization, it is adopted to seek for the maintenance action collection whose discounted long-term cost is the minimum.

    4. Numerical experiments of maintenance policy optimization

    Two numerical experiments were used to illustrate the proposed aero-engine life-cycle maintenance policy optimization algorithm in detail. According to the reinforcement learning framework,determinations of aero-engine states,maintenance actions, state transition matrices, and total cost matrices are described firstly.As traditional methods are unable to address the long service-life and the hybrid maintenance strategy synchronously,they were not adopted as the benchmark methods.The reinforcement learning algorithm of Gauss-Seidel value iteration was adopted in the experiments.

    4.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state should be determined firstly. As the multi-dimensional state space was adopted to represent the aero-engine state, performance, LLP, and random failure states were all considered in the first numerical experiment.

    The EGTM was adopted to represent the aero-engine performance state. For convenience, the EGTM time series were divided into several levels. In a sample fleet, there were three performance recovery actions, including minimal repair,medium repair, and overhaul repair. According to the performance recovery assumptions in Section 4.2,there should be at least five performance levels to fully illustrate the three performance recovery actions. In reinforcement learning, as more performance levels would make the optimization algorithm more complicated, five levels were able to present the aeroengine performance state. In the numerical experiment, the performance state was divided into five levels, denoted as{D1,D2,D3,D4,D5}, from good to bad, where D5denoted the worst performance level. Besides, the all-new performance state was denoted as D0.

    Because performance and random failure states were both transferred by probabilities, the random failure state was regarded as a specific‘‘performance state”.Thus,in the numerical experiment, performance and random failure states were represented by one state-space dimension, denoted as{D0,D1,D2,D3,D4,D5,F(xiàn)}.

    Although there were several LLPs in an aero-engine, for convenience,LLPs with the same life limitation were regarded as the same LLP type. In the first numerical experiment, one LLP type was taken into account,and the LLP state was measured by flight-cycles. Referring to the adopted performance state levels,the LLP state was divided into five levels,denoted as {T1,T2,T3,T4,T5}. Besides, T0denoted the all-new state.Thus, the aero-engine state was represented by a twodimensional state space, denoted as

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state F; Tj(j=0,1,2,3,4,5)denoted the LLP state. The performance and random failure states were denoted as the first dimension; the LLP state was denoted as the second dimension.

    Fig. 7 Schematic diagrams of performance recovery action effects on performance states.

    4.2. Maintenance actions

    In reinforcement learning, the aero-engine state was changed by performed maintenance actions, and maintenance actions should be determined according to the aero-engine state.Firstly, according to traditional and practical matters,46two performance recovery assumptions were made in the numerical experiment as follows:

    (1) An aero-engine could not be recovered to the all-new performance state by any performance recovery actions.

    (2) No maintenance action should be performed when the aero-engine was in the worst performance state.

    According to the sample fleet operation, three actions of minimal repair, medium repair, and overhaul repair were included in performance recovery actions. Thus, three performance recovery actions were adopted in the numerical experiment, denoted as Arec={Arec,1,Arec,2,Arec,3}, where Arec-h1was defined as the action to recover the performance state from Dxto Dx-1, when x-1 >0, or to keep in the performance state Dx,when x=1;Arec,2was defined as the action to recover the performance state from Dxto Dx-2, when x-2 >0, or to recover the performance state from Dxto D1,when 1 ≤x ≤2;Arec,3was defined as the action to recover the performance state from Dxto Dx-3, when x-3 >0, or to recover the performance state from Dxto D1, when 1 ≤x ≤3. Besides, Arec,0denoted no performance recovery action was performed.In the reinforcement learning framework, Fig. 7 shows performance recovery action effects on performance states.

    In the numerical experiment, the random failure state was regarded as a specific ‘‘performance state”. Thus, it was assumed that performance recovery actions could drive the random failure aero-engine back to the working state.

    As one LLP type was considered,LLP replacement actions were denoted as Arep={Arep,0,Arep,1}. Fig. 8 shows the LLP replacement action effect on LLP states.

    From the above, maintenance actions in the numerical experiment were represented by

    4.3. State transition

    Fig. 8 Schematic diagram of the LLP replacement action effect on LLP states.

    Based on the aforementioned methods, performance state Diand transition probability matrix P0were obtained from the sample fleet, as shown in

    Transition matrices of Arec,1, Arec,2, and Arec,3were P1, P2,and P3, denoted as

    where nllpdenoted the number of LLP types, and nrecadenoted the number of performance recovery actions.

    4.4. Maintenance policy optimization

    Based on the aforementioned methods, aero-engine states,maintenance actions, and state transition matrices were all determined. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy. The flow diagram of the proposed aero-engine maintenance policy optimization algorithm is shown in Fig. 9.

    Table 2 Transition probability matrix of an LLP replacement action.

    Fig. 9 Flow diagram of maintenance policy optimization.

    Because real cost data was unavailable, a hypothetical cost matrix was adopted in the numerical experiment. In engineering,the cost matrix may change according to the actual maintenance cost, and it would not distort the analysis of simulation results. In the numerical experiment, the LLP replacement cost Cllpwas assumed to be 350, while the performance recovery costs Crep,1, Crep,2, and Crep,3were assumed to be 300, 350, and 500. As an aero-engine has long service-life,future maintenance actions should be fully considered in lifecycle maintenance policy optimization.Thus,a larger discount factor γ=0.9 was adopted.In contrast to the Jacobi value iteration algorithm, the Gauss-Seidel value iteration algorithm has a faster convergence ability. Thus, in the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy.

    As the aero-engine state was represented by a twodimensional space, the optimal maintenance policy was presented by a two-dimensional policy map, shown in Fig. 10.

    Optimal maintenance actions on each decision epoch were shown in the policy map,and decision epochs were represented by aero-engine states. In the maintenance policy map of Fig. 10, the LLP state was regarded as the ordinate, and the performance state was regarded as the abscissa. Different maintenance actions were presented in different colors and shapes. In the legend, A0denoted action { Arep,0,Arec,0} ; A1denoted action { Arep,0,Arec,1} ; A2denoted action{ Arep,0,Arec,2} ;A4denoted action{ Arep,1,Arec,0} ;A6denoted action { Arep,1,Arec,2} . In engineering, a maintenance policy could be obtained according to the aero-engine state.

    Fig. 10 Maintenance policy map of the numerical experiment.

    4.5. Algorithm performance analysis

    In this section,a more complex numerical experiment was conducted to illustrate the proposed optimization algorithm with more detailed instructions. In engineering, an aero-engine is usually composed of more than one LLP type. Some LLP states were measured by flight-hour, different from the one in the first numerical experiment. Thus, a more complex numerical experiment with two LLP types was conducted.To distinguish the two numerical experiments,the first numerical experiment was named as Experiment 1,and the numerical experiment in this section was named as Experiment 2.

    Table 3 Transition matrices of no LLP replacement action.

    Table 4 Transition probability matrices of LLP1 replacement actions.

    4.5.1. Aero-engine states

    Two LLP types were considered in Experiment 2,and the two LLP states were measured by different units: one was flightcycle and the other was flight-hour.Thus,the aero-engine state was denoted as a three-dimensional state space. The three dimensions included the performance state,the LLP state represented by flight-cycle(LLP1), and the LLP state represented by flight-hour (LLP2).

    Corresponding to Experiment 1, in addition to all-new states T0and L0, the LLP1 state was divided into two levels,denoted as {T0,T1,T2};the LLP2 state was divided into three levels, denoted as {L0,L1,L2,L3}. Same as in Experiment 1,the performance state was represented by five levels, and the random failure state was regarded as a specific ‘‘performance state”. Thus, the aero-engine state space was represented by

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state; Tjdenoted the LLP1 state;Lkdenoted the LLP2 state.

    4.5.2. Maintenance actions and state transition

    In Experiment 2, LLP1 and LLP2 replacement actions were denoted as ArepT={ArepT,0,ArepT,1} and ArepL={ArepL,0,ArepL,1}. Performance recovery actions were the same as those in Experiment 1. Performance recovery assumptions were also available.Maintenance actions in Experiment 2 were represented by

    Similar to LLP1, the LLP2 state would transfer as the flight-hour increasing, and transition probabilities were unavailable for LLP2. However, aero-engine state transition matrices were changed by LLP2.

    On Em, action Am={ Arec,i,ArepT,0,ArepL,0|i=0,1,2,3}denoted no LLP replaced, and transition matrices are presented in Table 3, denoted as p(S,Arec,i,ArepT,0,ArepL,0|i=0,1,2,3)= p3{PF,i,,PN,PR,i|i=0,1,2,3}. The concrete forms of PF,, and PRwere the same as those in Experiment 1.

    State transition matrices of Am={ Arec,i,ArepT,1,ArepL,0|i=0,1,2,3} are presented in Table 4, denoted as p(S,Arec,i,ArepT,1,ArepL,0|i = 0,1,2,3) = p4{ PF,i,,PN,PR,i|i = 0,1,2,3} .

    State transition matrices of Am={Arec,i,ArepT,0,ArepL,1|i=0,1,2,3} are presented in Table 5, denoted as p(S,Arec,i,ArepT,0,ArepL,1|i = 0,1,2,3) = p5{PF,i,,PN,PR,i|i = 0,1,2,3}.

    State transition matrices of Am={Arec,i,ArepT,1,ArepL,1|i=0,1,2,3} are presented in Table 6, denoted as p(S,Arec,i,ArepT,1,ArepL,1|i = 0,1,2,3) = p6{PF,i,,PN,PR,i|i = 0,1,2,3}.

    4.5.3. Maintenance policy optimization

    In Experiment 2, the reinforcement learning algorithm of Gauss-Seidel value iteration was also adopted to optimize the maintenance policy. Hypothetical costs were adopted,and the LLP1 replacement cost Cllp,1was assumed to be 300 while the LLP2 replacement cost Cllp,2was assumed to be 600. The three performance recovery action costs were assumed to be 200, 500, and 800, respectively. The discount factor was set as γ=0.9. Because the aero-engine state was represented by a three-dimensional space, including the LLP1, LLP2, and performance states, the optimal maintenance policy was represented by a three-dimensional policy map, shown in Fig. 11.

    Table 6 Transition probability matrices of LLP1 and LLP2 replacement actions.

    Fig. 11 Maintenance policy map of Experiment 2.

    Table 7 Algorithm information of two experiments.

    In the three-dimensional policy map,the LLP1,LLP2,and performance states were regarded as x axis, z axis, and y axis,respectively.The maintenance actions were presented in different colors and shapes.In the legend of the maintenance policy map in Fig. 11, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action{Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action {Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.Based on the policy map, a maintenance policy was obtained according to the aero-engine state.

    4.5.4. Algorithm analysis

    In aforementioned two numerical experiments, a twodimensional state space was adopted in Experiment 1, while a three-dimensional state space was adopted in Experiment 2.It was obvious that state transition matrices of Experiment 2 were more complicated. Thus, the algorithm would become complex as the state number increased. The algorithm information of the two numerical experiments is shown in Table 7.

    In the aforementioned numerical experiments,performance and random failure states were defined as probability states,because they transferred by probabilities. LLP states did not transfer by probabilities, and were defined as definite states.As shown in the two numerical experiments, state transition matrix forms were impacted by definite states. Transition probabilities were impacted by probability states.In reinforcement learning, a larger state space would make the algorithm more complicated, and more iterations were needed to seek for the optimal policy. The impact of aero-engine state space complexity on the algorithm was obvious.

    As the discount factor is an important coefficient in reinforcement learning, the discount factor impact was analyzed by contrast experiments, which were based on Experiment 1.In addition to the discount factor, the other parameters were all the same as those in Experiment 1. Maintenance policy maps of discount factor analysis are shown in subgraphs of Fig. 12.

    Fig. 12 Policy maps of discount factor analysis in Example 1.

    In the policy maps,different maintenance actions were represented by different colors and shapes. In the legend of Fig. 12, A0denoted action {Arep,0,Arec,0}; A1denoted action{Arep,0,Arec,1}; A2denoted action {Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A6denoted action {Arep,1,Arec,2}. The analysis experiments showed that when the discount factor was set as γ >0.43,the optimal policy maps were the same as those in Fig. 10;when the discount factor was set as γ <0.35, the optimal policy maps were the same as those in Fig. 12(c). In Fig. 12, discount factors were set as γ = 0.43, γ = 0.41, and γ = 0.35,respectively. The analysis experiments showed that when a smaller discount factor was adopted, more low-cost maintenance actions were adopted in the optimal maintenance policy.It was consistent with the aforementioned analysis of the discount factor.

    As hypothetical maintenance costs were adopted in numerical experiments, cost impacts were analyzed. Based on the aforementioned experiments, the medium performance recovery cost was set as Crep,2=350, and it was regarded as the benchmark cost. Maintenance policy maps of different cost ratios are shown in subgraphs of Figs. 13 and 14.

    Fig. 13 Policy maps of performance recovery cost analysis in Example 1.

    In the legend of Fig. 13, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A3denoted action {Arep,0,Arec,3}; A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A7denoted action {Arep,1,Arec,3}. In the legend of Fig. 14, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A3denoted action {Arec,3,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A8denoted action {Arec,3,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1};A14denoted action{Arec,3,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.

    Fig. 14 Policy maps of performance recovery cost analysis in Example 2.

    Fig. 15 Policy maps of LLP replacement cost analysis in Example 1.

    As shown in Figs.13(a)and 14(a),showed the optimal policy when Crec,1decreased; Figs. 13(b) and 14(b) showed the optimal policy when Crec,3decreased; Figs. 13(c) and 14(c)showed the optimal policy when Crec,1and Crec,3decreased simultaneously. As shown in Figs. 13(c) and 14(c), when Crec,1and Crec,3decreased simultaneously, optimal policy changes were not obvious.

    The impact of the LLP replacement cost on the optimal maintenance policy was analyzed by contrast experiments.Experiment results showed that the optimal maintenance policy would not change as the LLP replacement cost increasing.However,LLP replacement action times would increase as the LLP replacement cost decreasing.Optimal policy maps of LLP replacement cost analysis are shown in Figs. 15-17.

    In the legend of Fig. 15, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A6denoted action {Arep,1,Arec,2}. In the legends of Figs. 16 and 17, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A2denoted action {Arec,2,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A7denoted action {Arec,2,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A13denoted action {Arec,2,ArepT,1,ArepL,1}; A15denoted action{Arec,0,ArepT,1,ArepL,1}.

    As shown in Fig. 15, the LLP replacement cost decreased from subgraph (a) to (c). In Figs. 16 and 17, LLP1 and LLP2 replacement costs decreased from subgraph (a) to (b).It was shown that LLP replacement times would increase as the LLP replacement cost decreasing, and policy changes appeared on interface decision epochs.

    In the aforementioned experiments, the impact of the LLP residual life was not considered. Thus, based on the assumption that the random failure probability would increase as the LLP residual life decreasing, contrast experiments were performed to analyze the optimization algorithm.Optimal policy maps are shown in Fig. 18.

    Fig. 16 Policy maps of LLP1 replacement cost analysis in Example 2.

    Fig. 17 Policy maps of LLP2 replacement cost analysis in Example 2.

    In the legend of Fig. 18, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}. Based on Experiment 2, according to the residual life of the elder LLP, random failure probabilities were enhanced by 5%, 10%, and 15% respectively from subgraph (a) to (c)of Fig. 18. Policy maps showed no variation. Thus, the LLP residual life may not affect the optimal maintenance policy.

    5. Conclusions

    Based on the reinforcement learning approach,an aero-engine life-cycle maintenance policy optimization algorithm was proposed, which was able to address the long service-life and the hybrid maintenance strategy synchronously. To address the hybrid maintenance strategy,the multi-dimensional state space was adopted to represent the aero-engine state. Based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the life-cycle maintenance policy.

    Compared with traditional optimization methods, the optimal maintenance policy was used to indicate when and how to repair an aero-engine, taking the place of maintenance intervals and work-scopes in traditional methods.Meanwhile, the aero-engine long service-life, the hybrid maintenance strategy, and random factor destabilization were all addressed by the proposed optimization algorithm.Because few historical data was available for training the pre-specified optimization model of the aero-engine lifecycle maintenance policy, the reinforcement learning approach provided an appropriate way. In the reinforcement learning framework, the aero-engine state space, maintenance actions, and state transition matrices were determined according to aero-engine real-life operation. The Gauss-Seidel value iteration algorithm was employed to solve the long-term decision-making problem. The proposed optimization algorithm would help in making a wiser aero-engine life-cycle maintenance policy, resulting in a lower life-cycle maintenance cost. Two numerical experiments and algorithm analyses were employed to illustrate the proposed optimization algorithm in detail.

    As real aero-engine maintenance cost data was unavailable,hypothetical data was adopted in the numerical experiments.In future studies, maintenance cost calculation methods deserve further attention to improve the applicability of the proposed optimization algorithm.

    Fig. 18 Policy maps with LLP lifetime impact on transition probability.

    Acknowledgments

    The authors thank anonymous reviewers for their critical and constructive review of the manuscript. This work was cosupported by the Key National Natural Science Foundation of China (No. U1533202), the Civil Aviation Administration of China (No. MHRD20150104), and the Shandong Independent Innovation and Achievements Transformation Fund,China (No. 2014CGZH1101).

    亚洲国产欧美网| 精品卡一卡二卡四卡免费| 国产精品1区2区在线观看.| 每晚都被弄得嗷嗷叫到高潮| 他把我摸到了高潮在线观看| 99久久无色码亚洲精品果冻| 国产单亲对白刺激| 首页视频小说图片口味搜索| 亚洲 欧美一区二区三区| 国产精品免费视频内射| 欧美黑人精品巨大| 色播亚洲综合网| 久久九九热精品免费| 亚洲avbb在线观看| 母亲3免费完整高清在线观看| 一进一出好大好爽视频| 欧美zozozo另类| 黄色片一级片一级黄色片| 国产在线精品亚洲第一网站| 精品不卡国产一区二区三区| 婷婷六月久久综合丁香| 夜夜爽天天搞| 欧美黄色淫秽网站| 欧美绝顶高潮抽搐喷水| 国产成人系列免费观看| 淫秽高清视频在线观看| 三级毛片av免费| 国产又黄又爽又无遮挡在线| 成人一区二区视频在线观看| netflix在线观看网站| 欧美绝顶高潮抽搐喷水| netflix在线观看网站| av电影中文网址| 国产亚洲精品一区二区www| 少妇熟女aⅴ在线视频| 婷婷精品国产亚洲av| 国产亚洲精品久久久久久毛片| 亚洲三区欧美一区| 少妇粗大呻吟视频| 99国产极品粉嫩在线观看| 久久人人精品亚洲av| 精品国产一区二区三区四区第35| 亚洲av电影在线进入| 亚洲av日韩精品久久久久久密| 亚洲自偷自拍图片 自拍| 欧美乱码精品一区二区三区| 久久精品亚洲精品国产色婷小说| 国内揄拍国产精品人妻在线 | 大型黄色视频在线免费观看| 999久久久国产精品视频| av天堂在线播放| 亚洲国产精品sss在线观看| 男女下面进入的视频免费午夜 | 亚洲国产欧美一区二区综合| 国产色视频综合| 日本撒尿小便嘘嘘汇集6| 特大巨黑吊av在线直播 | 久久亚洲精品不卡| 免费在线观看完整版高清| 美女高潮到喷水免费观看| 男人操女人黄网站| 久久国产亚洲av麻豆专区| 一进一出抽搐gif免费好疼| 国产精品二区激情视频| 国产亚洲av嫩草精品影院| 久久人妻av系列| 777久久人妻少妇嫩草av网站| 99re在线观看精品视频| 欧美成人性av电影在线观看| 日本一区二区免费在线视频| 国产精品 欧美亚洲| 亚洲国产精品合色在线| 午夜免费观看网址| 国产精品av久久久久免费| 母亲3免费完整高清在线观看| 欧美日韩精品网址| 少妇粗大呻吟视频| 人人妻,人人澡人人爽秒播| 男女下面进入的视频免费午夜 | 啪啪无遮挡十八禁网站| 俺也久久电影网| 精品欧美国产一区二区三| 免费在线观看亚洲国产| 美女扒开内裤让男人捅视频| 精品一区二区三区av网在线观看| 黄色丝袜av网址大全| 精品一区二区三区av网在线观看| av电影中文网址| 亚洲精品国产区一区二| 天堂动漫精品| 美女 人体艺术 gogo| 亚洲国产欧美网| 欧美又色又爽又黄视频| 国产成人精品无人区| 精品久久蜜臀av无| 老司机深夜福利视频在线观看| 波多野结衣av一区二区av| 久久中文字幕一级| 成人三级做爰电影| 国产精品亚洲一级av第二区| 香蕉av资源在线| 国产成人影院久久av| 欧美大码av| 亚洲熟女毛片儿| 宅男免费午夜| 亚洲最大成人中文| 国产三级在线视频| 啪啪无遮挡十八禁网站| 亚洲第一电影网av| 日韩欧美 国产精品| 国产精品九九99| www.www免费av| 亚洲av电影不卡..在线观看| 又紧又爽又黄一区二区| 亚洲成av片中文字幕在线观看| 国产色视频综合| 久久热在线av| 久热爱精品视频在线9| 精品人妻1区二区| 国产亚洲欧美在线一区二区| 国产v大片淫在线免费观看| 天天一区二区日本电影三级| 亚洲片人在线观看| 午夜精品久久久久久毛片777| 99精品在免费线老司机午夜| 国产精品久久久av美女十八| 这个男人来自地球电影免费观看| 免费一级毛片在线播放高清视频| 亚洲国产毛片av蜜桃av| www.www免费av| 国产高清videossex| 日韩国内少妇激情av| 久久国产乱子伦精品免费另类| 欧美激情高清一区二区三区| 亚洲欧美精品综合久久99| 美女国产高潮福利片在线看| 天堂动漫精品| 免费在线观看亚洲国产| 又黄又粗又硬又大视频| 国产不卡一卡二| 丝袜人妻中文字幕| 亚洲avbb在线观看| 757午夜福利合集在线观看| 最好的美女福利视频网| 国产精品 国内视频| 国内少妇人妻偷人精品xxx网站 | 午夜福利欧美成人| 两性夫妻黄色片| 看片在线看免费视频| 自线自在国产av| 亚洲无线在线观看| 午夜久久久在线观看| 一级黄色大片毛片| 国产黄a三级三级三级人| 热99re8久久精品国产| 99久久99久久久精品蜜桃| 村上凉子中文字幕在线| 久久性视频一级片| 成人欧美大片| 波多野结衣av一区二区av| 啪啪无遮挡十八禁网站| 制服人妻中文乱码| 国产精品99久久99久久久不卡| 啦啦啦免费观看视频1| 麻豆av在线久日| 久久人妻av系列| 男人舔女人的私密视频| 成在线人永久免费视频| 免费在线观看亚洲国产| 亚洲专区国产一区二区| 深夜精品福利| 久久久精品国产亚洲av高清涩受| 99国产精品一区二区三区| 亚洲片人在线观看| bbb黄色大片| 亚洲av片天天在线观看| 国产蜜桃级精品一区二区三区| 精品欧美国产一区二区三| 精品国产国语对白av| 日韩 欧美 亚洲 中文字幕| 视频在线观看一区二区三区| 99久久99久久久精品蜜桃| 欧美不卡视频在线免费观看 | 每晚都被弄得嗷嗷叫到高潮| 欧美日韩亚洲综合一区二区三区_| 校园春色视频在线观看| 久久婷婷成人综合色麻豆| 亚洲av日韩精品久久久久久密| 成人国语在线视频| 999精品在线视频| 免费高清在线观看日韩| 久99久视频精品免费| 久久青草综合色| 久久久国产欧美日韩av| 午夜免费激情av| 国产激情欧美一区二区| 欧美av亚洲av综合av国产av| 在线观看午夜福利视频| 色播在线永久视频| 亚洲精品久久成人aⅴ小说| 在线观看免费日韩欧美大片| 亚洲午夜理论影院| 国产精品香港三级国产av潘金莲| 最近在线观看免费完整版| 精品国产国语对白av| 日韩欧美一区视频在线观看| aaaaa片日本免费| 欧美日韩精品网址| 黄色视频,在线免费观看| 校园春色视频在线观看| 午夜视频精品福利| 丝袜人妻中文字幕| 免费在线观看视频国产中文字幕亚洲| 亚洲精品色激情综合| 亚洲成a人片在线一区二区| 亚洲专区国产一区二区| 叶爱在线成人免费视频播放| 亚洲av电影在线进入| 少妇的丰满在线观看| 久久国产精品影院| 欧美+亚洲+日韩+国产| 可以在线观看毛片的网站| 狠狠狠狠99中文字幕| 亚洲av第一区精品v没综合| 精品熟女少妇八av免费久了| 欧美国产日韩亚洲一区| 少妇熟女aⅴ在线视频| 久久国产乱子伦精品免费另类| 国产一区二区三区在线臀色熟女| 好男人在线观看高清免费视频 | 999久久久国产精品视频| 黄色视频不卡| av福利片在线| 国产精品电影一区二区三区| 不卡一级毛片| 国产精品一区二区三区四区久久 | 淫妇啪啪啪对白视频| 精品福利观看| 他把我摸到了高潮在线观看| 男人操女人黄网站| 久久精品亚洲精品国产色婷小说| 美女免费视频网站| 亚洲欧美一区二区三区黑人| 丰满人妻熟妇乱又伦精品不卡| 好男人电影高清在线观看| 一级片免费观看大全| 成人欧美大片| 亚洲男人天堂网一区| 国产区一区二久久| 日韩欧美 国产精品| 免费人成视频x8x8入口观看| 黄色丝袜av网址大全| 国产午夜福利久久久久久| 久久热在线av| 国产亚洲欧美98| 欧美日本视频| 哪里可以看免费的av片| 熟女少妇亚洲综合色aaa.| 欧美日本视频| 亚洲av美国av| 18禁黄网站禁片午夜丰满| 黄色毛片三级朝国网站| 国产又色又爽无遮挡免费看| 国产亚洲欧美98| 欧美日韩黄片免| 69av精品久久久久久| 成人欧美大片| 他把我摸到了高潮在线观看| 国产成人欧美在线观看| av天堂在线播放| 成年人黄色毛片网站| 亚洲一区二区三区不卡视频| videosex国产| 日日爽夜夜爽网站| 久久精品aⅴ一区二区三区四区| 男人的好看免费观看在线视频 | 999精品在线视频| 亚洲aⅴ乱码一区二区在线播放 | 亚洲国产高清在线一区二区三 | 国产v大片淫在线免费观看| 精品国产超薄肉色丝袜足j| 久久精品国产亚洲av高清一级| 日日摸夜夜添夜夜添小说| 美女免费视频网站| 一区二区三区精品91| 国产伦在线观看视频一区| 他把我摸到了高潮在线观看| 欧美色视频一区免费| a级毛片a级免费在线| 欧美日韩一级在线毛片| 麻豆一二三区av精品| 男人舔奶头视频| 亚洲av熟女| 国产精品一区二区免费欧美| 日本五十路高清| 亚洲七黄色美女视频| 精品第一国产精品| 韩国精品一区二区三区| 免费女性裸体啪啪无遮挡网站| 国产精品久久电影中文字幕| 白带黄色成豆腐渣| 久久狼人影院| 国产精品久久久久久人妻精品电影| 久久草成人影院| 757午夜福利合集在线观看| 久久精品国产亚洲av香蕉五月| 久久精品成人免费网站| 人妻丰满熟妇av一区二区三区| 97碰自拍视频| 精品免费久久久久久久清纯| 国产精品 欧美亚洲| 亚洲中文字幕一区二区三区有码在线看 | 亚洲中文字幕一区二区三区有码在线看 | 国产三级黄色录像| x7x7x7水蜜桃| 精品一区二区三区av网在线观看| 激情在线观看视频在线高清| 国产高清视频在线播放一区| 久久亚洲精品不卡| videosex国产| 1024手机看黄色片| 高清在线国产一区| 欧美日本亚洲视频在线播放| 亚洲第一av免费看| 亚洲欧美日韩无卡精品| 国产精品日韩av在线免费观看| 欧美成人性av电影在线观看| 麻豆成人午夜福利视频| 精品久久蜜臀av无| 啦啦啦韩国在线观看视频| 欧美日韩精品网址| 一区二区三区高清视频在线| 欧美日韩亚洲国产一区二区在线观看| 国产亚洲欧美精品永久| 亚洲 欧美一区二区三区| 国产一级毛片七仙女欲春2 | e午夜精品久久久久久久| 少妇熟女aⅴ在线视频| 成人国产一区最新在线观看| 久久久精品欧美日韩精品| 午夜福利在线观看吧| 好看av亚洲va欧美ⅴa在| 一进一出好大好爽视频| 女警被强在线播放| 搞女人的毛片| 国内揄拍国产精品人妻在线 | 精品欧美一区二区三区在线| 国产乱人伦免费视频| 长腿黑丝高跟| 久久精品国产99精品国产亚洲性色| 黄色成人免费大全| 久久久久久国产a免费观看| 亚洲成国产人片在线观看| 女同久久另类99精品国产91| avwww免费| 中文字幕人妻熟女乱码| 熟妇人妻久久中文字幕3abv| 国产成人系列免费观看| 高清毛片免费观看视频网站| 日日摸夜夜添夜夜添小说| 在线观看一区二区三区| 在线观看免费午夜福利视频| 欧美精品亚洲一区二区| 国产精品爽爽va在线观看网站 | 757午夜福利合集在线观看| 亚洲av美国av| 非洲黑人性xxxx精品又粗又长| 亚洲va日本ⅴa欧美va伊人久久| 男女做爰动态图高潮gif福利片| 别揉我奶头~嗯~啊~动态视频| 国产免费男女视频| 国产私拍福利视频在线观看| e午夜精品久久久久久久| 日韩中文字幕欧美一区二区| 国产一区二区三区在线臀色熟女| 丰满人妻熟妇乱又伦精品不卡| 男人舔奶头视频| 中亚洲国语对白在线视频| 欧美精品啪啪一区二区三区| 亚洲第一av免费看| 禁无遮挡网站| 国产熟女xx| 又紧又爽又黄一区二区| 一二三四社区在线视频社区8| 很黄的视频免费| 一级a爱视频在线免费观看| 啦啦啦 在线观看视频| 精品久久蜜臀av无| 悠悠久久av| 人成视频在线观看免费观看| 国产激情欧美一区二区| 亚洲片人在线观看| 美女 人体艺术 gogo| 久久国产精品影院| 国产精品九九99| 一卡2卡三卡四卡精品乱码亚洲| 亚洲中文字幕一区二区三区有码在线看 | 又大又爽又粗| 熟女电影av网| 两个人视频免费观看高清| 一级毛片高清免费大全| 国产精品一区二区三区四区久久 | 国产午夜福利久久久久久| 1024视频免费在线观看| 黄网站色视频无遮挡免费观看| 欧美激情高清一区二区三区| 99精品在免费线老司机午夜| 丁香欧美五月| 一级作爱视频免费观看| 久久久久国内视频| 亚洲av第一区精品v没综合| 国产1区2区3区精品| 国产不卡一卡二| 桃色一区二区三区在线观看| 男人舔奶头视频| 久久午夜亚洲精品久久| 久久香蕉精品热| 久久久久久久精品吃奶| 亚洲午夜理论影院| 欧美黑人欧美精品刺激| 精品欧美一区二区三区在线| 亚洲av成人不卡在线观看播放网| 亚洲成av人片免费观看| 国产成+人综合+亚洲专区| 免费高清视频大片| 成人18禁高潮啪啪吃奶动态图| 午夜福利欧美成人| xxxwww97欧美| 亚洲电影在线观看av| 丝袜美腿诱惑在线| 色综合婷婷激情| 黄色视频不卡| 日韩欧美国产在线观看| www日本黄色视频网| 免费在线观看成人毛片| 身体一侧抽搐| 国产精品久久视频播放| 亚洲国产欧美网| 国产午夜精品久久久久久| 久久午夜综合久久蜜桃| 啦啦啦免费观看视频1| 美女 人体艺术 gogo| 一区二区三区激情视频| 一个人观看的视频www高清免费观看 | 久久午夜综合久久蜜桃| 久久精品aⅴ一区二区三区四区| 国产国语露脸激情在线看| 一级黄色大片毛片| 久久中文字幕一级| 精品久久久久久久久久免费视频| 18禁黄网站禁片免费观看直播| 亚洲人成电影免费在线| 夜夜看夜夜爽夜夜摸| 亚洲五月天丁香| 不卡av一区二区三区| 久久人妻福利社区极品人妻图片| 一级毛片精品| 国产精品一区二区三区四区久久 | 变态另类成人亚洲欧美熟女| 日本a在线网址| 国产亚洲精品一区二区www| 亚洲中文av在线| 国内精品久久久久精免费| 免费在线观看视频国产中文字幕亚洲| 国产在线精品亚洲第一网站| 亚洲真实伦在线观看| 在线观看免费日韩欧美大片| 欧美 亚洲 国产 日韩一| 变态另类丝袜制服| 一级毛片精品| 黄频高清免费视频| 一二三四在线观看免费中文在| 免费看十八禁软件| 亚洲成人久久性| 日韩有码中文字幕| 麻豆成人av在线观看| 麻豆av在线久日| 999久久久精品免费观看国产| 国产极品粉嫩免费观看在线| 日本五十路高清| 欧美日韩亚洲国产一区二区在线观看| 亚洲第一青青草原| 久久 成人 亚洲| 中文字幕另类日韩欧美亚洲嫩草| 久热爱精品视频在线9| 禁无遮挡网站| 99国产极品粉嫩在线观看| 首页视频小说图片口味搜索| 成人三级黄色视频| 性欧美人与动物交配| 男人舔女人下体高潮全视频| 伦理电影免费视频| 亚洲久久久国产精品| 黄色毛片三级朝国网站| 久久久水蜜桃国产精品网| 国内精品久久久久精免费| 中文字幕人妻丝袜一区二区| 中文在线观看免费www的网站 | 听说在线观看完整版免费高清| netflix在线观看网站| 亚洲国产欧洲综合997久久, | 国产成年人精品一区二区| 在线播放国产精品三级| 亚洲第一av免费看| 国产单亲对白刺激| 国产高清视频在线播放一区| 久久精品国产亚洲av高清一级| 九色国产91popny在线| 亚洲 国产 在线| 少妇裸体淫交视频免费看高清 | 欧美激情 高清一区二区三区| 日韩有码中文字幕| 一区二区三区国产精品乱码| 成人三级黄色视频| 丝袜人妻中文字幕| 热re99久久国产66热| 国产不卡一卡二| 久9热在线精品视频| 成人永久免费在线观看视频| 免费av毛片视频| 国产成人一区二区三区免费视频网站| 国产精品国产高清国产av| 国产成人系列免费观看| 制服诱惑二区| 国产精品乱码一区二三区的特点| 久久久久久大精品| 中文资源天堂在线| 精品免费久久久久久久清纯| 成人国产综合亚洲| 免费搜索国产男女视频| 国产乱人伦免费视频| 久久精品91无色码中文字幕| 熟女电影av网| 99久久国产精品久久久| 久热这里只有精品99| 国产1区2区3区精品| cao死你这个sao货| 日本免费一区二区三区高清不卡| 日本一区二区免费在线视频| 国产av在哪里看| 在线播放国产精品三级| 亚洲人成77777在线视频| 老司机在亚洲福利影院| 黄色片一级片一级黄色片| 国产不卡一卡二| 国产精品电影一区二区三区| 免费看美女性在线毛片视频| 一边摸一边做爽爽视频免费| 桃色一区二区三区在线观看| 精品乱码久久久久久99久播| 91在线观看av| 高清毛片免费观看视频网站| 午夜福利18| 人妻久久中文字幕网| 久久99热这里只有精品18| 黄片播放在线免费| 亚洲熟妇熟女久久| 香蕉久久夜色| 亚洲成av人片免费观看| 国产精品国产高清国产av| 免费看十八禁软件| 亚洲中文字幕一区二区三区有码在线看 | 精品一区二区三区av网在线观看| 99精品欧美一区二区三区四区| 免费一级毛片在线播放高清视频| 露出奶头的视频| 精品卡一卡二卡四卡免费| 欧美日韩乱码在线| 亚洲色图av天堂| 日韩高清综合在线| 亚洲 国产 在线| 精品日产1卡2卡| 国产精品免费一区二区三区在线| 国产激情偷乱视频一区二区| 搡老妇女老女人老熟妇| 侵犯人妻中文字幕一二三四区| 国产欧美日韩一区二区精品| 国产野战对白在线观看| 欧洲精品卡2卡3卡4卡5卡区| 欧美激情 高清一区二区三区| 窝窝影院91人妻| 制服丝袜大香蕉在线| 久久国产精品影院| 国产av又大| 亚洲av成人不卡在线观看播放网| 精品欧美一区二区三区在线| 国产私拍福利视频在线观看| 精品久久久久久久人妻蜜臀av| 国产精品精品国产色婷婷| 日韩国内少妇激情av| 十八禁人妻一区二区| 日本免费一区二区三区高清不卡| www国产在线视频色| 亚洲激情在线av| 日韩大尺度精品在线看网址| 国产精品日韩av在线免费观看| 国产亚洲精品一区二区www| 欧美在线一区亚洲| 动漫黄色视频在线观看| 禁无遮挡网站| 男人舔奶头视频| 亚洲精品国产一区二区精华液| 欧美黄色片欧美黄色片| 亚洲午夜理论影院| 啪啪无遮挡十八禁网站| 日本黄色视频三级网站网址| 黄色片一级片一级黄色片| 美女午夜性视频免费| 亚洲天堂国产精品一区在线| 中文字幕人妻丝袜一区二区| 亚洲av片天天在线观看| 亚洲七黄色美女视频| 国产精品一区二区精品视频观看| 日韩高清综合在线| 91字幕亚洲|