• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach

    2019-10-26 01:12:16ZhenLIShishengZHONGLinLIN
    CHINESE JOURNAL OF AERONAUTICS 2019年9期

    Zhen LI, Shisheng ZHONG, Lin LIN

    School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China

    KEYWORDS Aero-engine;Hybrid strategy;Maintenance policy;Optimization algorithm;Reinforcement learning

    Abstract An aero-engine maintenance policy plays a crucial role in reasonably reducing maintenance cost. An aero-engine is a type of complex equipment with long service-life. In engineering,a hybrid maintenance strategy is adopted to improve the aero-engine operational reliability. Thus,the long service-life and the hybrid maintenance strategy should be considered synchronously in aero-engine maintenance policy optimization. This paper proposes an aero-engine life-cycle maintenance policy optimization algorithm that synchronously considers the long service-life and the hybrid maintenance strategy. The reinforcement learning approach was adopted to illustrate the optimization framework, in which maintenance policy optimization was formulated as a Markov decision process. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the maintenance policy. Compared with traditional aero-engine maintenance policy optimization methods,the long service-life and the hybrid maintenance strategy could be addressed synchronously by the proposed algorithm. Two numerical experiments and algorithm analyses were performed to illustrate the optimization algorithm in detail.

    1. Introduction

    An aero-engine, which is composed of mechanic-electrichydraulic coupling systems,is the power plant of an aircraft.1,2It has been reported that more than 30% of aircraft mechanical problems are related to aero-engines, and the aero-engine maintenance cost contributes to about 30% of an airline's direct operating cost.3A maintenance optimization method provides an available way to reduce the maintenance cost reasonably.4In general, excessive maintenance is costly, while insufficient maintenance may lead to disasters.Thus,a maintenance policy plays a crucial role in balancing the maintenance cost and operational reliability.5However, it is not an easy work to optimize an aero-engine maintenance policy manually,especially taking the long service-life and the hybrid maintenance strategy into consideration synchronously.

    In engineering,a hybrid maintenance strategy is adopted to improve the civil aero-engine operational reliability.Strategies of Condition-Based Maintenance (CBM), Hard-time Maintenance (HM), and failure Corrective Maintenance (CM) are included in the hybrid maintenance strategy.6As aero-engine performance deterioration is inevitable,7gas path performance parameters are monitored for CBM. The Life Limit Part(LLP) should be replaced before its life limitation,8and HM is adopted. Moreover, CM is performed when an aeroengine is in the random failure state.Thus,the hybrid maintenance strategy should be considered in aero-engine maintenance policy optimization. However, few existing aero-engine maintenance optimization methods are able to address the hybrid strategy. At the same time, an aero-engine is a type of equipment with long service-life,9which should be synchronously considered in maintenance policy optimization.To the best of our knowledge, this is the first paper to investigate an aero-engine maintenance policy optimization method that can synchronously address the hybrid maintenance strategy and long service-life.

    In traditional aero-engine maintenance optimization methods, maintenance intervals are used to decide when to repair an aero-engine, and maintenance work-scopes indicate how to carry out maintenance actions. Maintenance intervals and work-scopes can be obtained by traditional separate models.For example, LLP replacement intervals and performance recovery intervals can be optimized separately by current models.10-12Based on maintenance intervals, maintenance workscopes are obtained by traditional decision-making models.13,14Based on traditional optimization methods, an aeroengine maintenance decision support system was proposed by Fu et al.,15in which maintenance interval and work-scope optimization models were presented. An optimization method for reliability-centered maintenance was proposed by Crocker and Kumar,16in which the concepts of soft life and hard life were used to optimize a military aero-engine maintenance policy.A multi-objective evolutionary algorithm was also adopted to solve the aero-engine maintenance scheduling problem,17taking the module exchange into consideration. To optimize an aero-engine maintenance policy, traditional optimization methods would become extremely complicated when the long service-life and the hybrid maintenance strategy are considered synchronously.18

    In general, machine learning methods include supervised learning, unsupervised learning, and reinforcement learning,19and the reinforcement learning method has attracted increasing interests in solving decision-making problems.20Reinforcement learning represents a machine learning method in which an agent learns how to behave through action rewards.Different from widely used supervised learning methods, there is no presentation of input and output pairs in reinforcement learning. In reinforcement learning, an agent chooses an available action according to the environment on each decision epoch.The chosen action changes the environment, along with a reward to the agent. The objective of the agent is to find the action collection, whose reward is maximal in the long run.

    Reinforcement learning methods have been adopted in energy system charging policy optimization,21,22energy system and distributed system schedule determination,23,24multiple robotic task optimization,25demand response optimization,26,27robust control optimization,28multiple satellites task planning,29et al. Although reinforcement learning methods have been successfully applied, they have not aroused much attention in aero-engine maintenance policy optimization. Reinforcement learning does provide a more appropriate way for aero-engine life-cycle maintenance policy optimization. Thus, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on reinforcement learning. The main contributions of this paper are as follows:

    (1) An aero-engine life-cycle maintenance policy optimization algorithm is proposed, which can synchronously address the aero-engine long service-life and the hybrid maintenance strategy.

    (2) To address the hybrid strategy, the aero-engine state is represented by a multi-dimensional state space.

    (3) Reinforcement learning is adopted to illustrate the maintenance policy optimization. In the reinforcement learning framework, maintenance policy optimization is formulated as a discrete Markov Decision Process(MDP), and the Gauss-Seidel value iteration algorithm is adopted to optimize the maintenance policy.

    The remainder of this paper is organized as follows.Section 2 introduces aero-engine maintenance policy optimization and the reinforcement learning approach, and the deficiencies of traditional optimization methods are analyzed.In Section 3, the proposed aero-engine life-cycle maintenance policy optimization algorithm is described in detail. In Section 4, two simulation experiments and algorithm analysis are used to illustrate the proposed optimization algorithm.Conclusions and future work are discussed in Section 5.

    2. Aero-engine maintenance policy optimization and reinforcement learning

    2.1. Aero-engine maintenance policy optimization

    A maintenance policy indicates when and how to repair an aero-engine. Traditionally, maintenance intervals indicate when to repair an aero-engine, which are obtained by traditional separate optimization models, addressing CBM, HM and,CM strategies separately.In traditional methods,maintenance work-scopes indicate how to repair an aero-engine,which are optimized based on the outputs of interval optimization models. In summary, there are several deficiencies in traditional aero-engine maintenance optimization methods, as follows:

    (1) In traditional methods,maintenance intervals and workscopes are optimized by separate models, and workscopes are optimized based on interval optimization results. Thus, the interactions of maintenance intervals and work-scopes are neglected.Moreover,interval optimization errors would be propagated to work-scope optimization.

    (2) Because the hybrid maintenance strategy is adopted for civil aero-engines, hybrid strategies should be addressed synchronously in optimization. However, traditional optimization methods address hybrid strategies separately, and interactions of hybrid strategies are neglected.

    (3) It is difficult for traditional optimization methods to address the aero-engine long-service life and the hybrid maintenance strategy synchronously.

    (4) Definite optimization results are obtained by traditional optimization methods. Due to random factors, definite optimization results may be poorly applicable in engineering.

    To deal with the aforementioned deficiencies of traditional methods, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on the reinforcement learning approach. Taking the place of maintenance intervals and work-scopes in traditional methods, the maintenance policy indicates when and how to repair an aero-engine.The proposed optimization algorithm is able to address the hybrid maintenance strategy and long service-life synchronously.

    To address the hybrid maintenance strategy, a multidimensional space is adopted to represent the aero-engine state.The maintenance strategies of CBM and CM are formulated as an MDP,and imperfect repair and random factors are all considered in state transition. Aero-engine maintenance policy optimization is illustrated by the reinforcement learning approach, and the Gauss-Seidel value iteration algorithm is adopted to seek for the optimal maintenance policy. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm makes it available to optimize the lifecycle maintenance policy. A comparison between traditional optimization methods and the proposed optimization algorithm is shown in Fig. 1.

    In Fig.1,traditional aero-engine maintenance optimization methods are shown on the left.Traditional methods are constituted by separate models,including an LLP interval optimization model, a performance interval optimization model, and a work-scope optimization model.LLP replacement and performance recovery intervals are obtained by separate optimization models. Based on the LLP replacement, performance recovery, and corrective maintenance intervals, the maintenance work-scope is optimized. The proposed aero-engine life-cycle maintenance policy optimization algorithm is shown on the right in Fig. 1. Based on the reinforcement learning framework, the strategies of HM, CM, and CBM are addressed synchronously by the proposed optimization algorithm. The traditional separate optimization models of the LLP replacement interval, the performance recovery interval,and the maintenance work-scope are replaced by the proposed optimization algorithm.

    2.2. Reinforcement learning approach

    Reinforcement learning is a machine learning method,which is widely used to solve multi-step, sequential-decision problems.Different from supervised learning, no pre-specified model is required in reinforcement learning. In aero-engine life-cycle maintenance policy optimization, few historical data is available for training the pre-specified mode. Thus, reinforcement learning provides a more appropriate way for aero-engine life-cycle maintenance policy optimization. Meanwhile, the aero-engine long service-life and the hybrid maintenance strategy can be addressed synchronously by reinforcement learning.

    In reinforcement learning, an agent takes on the work of optimizing an aero-engine maintenance policy. The agent is able to respond to dynamically changing aero-engine states through ongoing learning methods.30In aero-engine maintenance policy optimization, aero-engine states are represented by a multi-dimensional state space. To optimize the maintenance policy, the agent chooses a maintenance action according to the aero-engine state. The aero-engine state is changed by the chosen maintenance action,along with the maintenance cost, as shown in Fig. 2. The optimal objective of the agent is to find the maintenance action collection, whose total cost is the minimum in the long run.

    Fig. 2 Schematic diagram of reinforcement learning.

    Value iteration is a reinforcement learning algorithm that is widely adopted in solving decision-making problems. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm provides an appropriate way to optimize an aero-engine life-cycle maintenance policy. According to the Gauss-Seidel value iteration algorithm, an agent would run multiple episodes for the purpose of exploring and finding the optimal policy.The learning process is conducted for a sufficient number of iterations,and the total cost of each iteration is recorded.The minimum total cost is represented as Q-value,which is updated every iteration, and the Bellman equation is adopted as the updating mechanism in the Gauss-Seidel value iteration algorithm. The convergence of value iterative methods has been widely proven.31Thus, based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm.

    3. Aero-engine life-cycle maintenance policy optimization algorithm

    The reinforcement learning approach is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm, in which the hybrid maintenance strategy and long service-life are considered synchronously.In the reinforcement learning framework, the aero-engine state, maintenance actions, state transition matrices, and maintenance costs should be determined. To address the hybrid maintenance strategy, a multi-dimensional state space is adopted to represent the aero-engine state, taking performance, LLP, and random failure states into consideration synchronously. The optimal objective of the proposed optimization algorithm is to obtain a maintenance policy whose total cost is the minimum in the long run. The reinforcement learning algorithm of Gauss-Seidel value iteration is adopted, addressing longterm optimization. In this section, the proposed optimization algorithm is described in detail.sub-state Yjdenotes the performance state, and Zkdenotes the random failure state.

    (1) LLP state

    In engineering,an LLP must be replaced before its life limitation,and the HM strategy is adopted for LLP maintenance.Traditionally, the flight-cycle or flight-hour is adopted to represent the LLP state. For example, a fan shaft is an LLP of a CFM56-5B aero-engine, and its life limitation is 30,000 flightcycles. That is, a fan shaft must be replaced before 30,000 flight-cycles. For convenience, the LLP state is represented by discrete time increments.32In the proposed optimization algorithm, the LLP state is represented by several state levels,denoted as {Ti|i=0,1,2,...}, where T0denotes the all-new state; when m <n, state Tnis ‘‘older” than state Tm. It is defined that an LLP is in state Tn, when tn-1<tllp≤tn; tn-1and tnare boundary values; tllpis an LLP real state, measured by flight-cycle or flight-hour.

    (2) Performance state

    As the CBM strategy is adopted in aero-engine maintenance, the aero-engine performance state should also be considered in the proposed optimization algorithm. Most of the in-service aero-engines are equipped with condition monitoring systems, and performance parameters are sent to ground by the aircraft communication addressing and reporting system in quasi real-time. Traditionally, the aero-engine performance state is assessed by performance parameters.33,34In engineering, the Exhaust Gas Temperature Margin (EGTM)is adopted as an aero-engine performance indicator.35The EGTM is defined as the margin temperature between exhaust gas temperature and red-line temperature. The aero-engine performance degrades as it operates, presented as EGTM declining.36,37When the EGTM is close to the limitation, performance recovery maintenance should be performed. The 1000 flight-cycles EGTM parameters of a CFM56-5B aeroengine are shown in Fig. 3.

    As EGTM parameters are typical time series,38it would make the algorithm extremely complicated when EGTM time series are adopted directly.For convenience,the CBM strategy is formulated as a discrete MDP in the reinforcement learning framework. Thus, the aero-engine performance state is represented by several levels, denoted as {Di|i=0,1,2,...}, where D0denotes the all-new performance state; when m <n, state Dnis worse than state Dm. It is defined that the performance state is Dn, when dn-1<dper≤dn, in which dn-1and dnare

    3.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state is changed by performed actions. As the hybrid maintenance strategy is considered in the proposed optimization algorithm,a multi-dimensional state space is adopted to represent the aero-engine state, in which performance, LLP, and random failure states are considered synchronously. The multidimensional state space is represented by S={Xi,Yj,Zk,...},where Xi,Yj,Zk,... denote the sub-states in the multidimensional state space. Each sub-state denotes one considered factor, for example, sub-state Xidenotes the LLP state,the boundary values;dperis the EGTM real value,measured by centigrade. Thus, in the proposed optimization algorithm, the aero-engine performance state is indicated by the performance levels.

    Fig. 3 1000 flight-cycles EGTM of a CFM56-5B aero-engine.

    (3) Random failure state

    Although an aero-engine is with high reliability, it is subjected to random failures in practice. When an aero-engine is in the random failure state, CM should be performed to drive it back to the working state. Unlike the LLP or performance state, the random failure state is represented by two levels,denoted as {F0,F(xiàn)1}, where F0denotes the working state and F1denotes the random failure state.

    From the above,when LLP,performance,and random failure states are all considered, the aero-engine state is represented by a multi-dimensional state space, denoted as

    where Tidenotes the LLP state, Djdenotes the performance state, and Fkdenotes the random failure state.

    3.2. Maintenance actions

    In reinforcement learning, maintenance actions are performed on every decision epoch. The decision epoch is denoted as{Eii=| 0,1,2,...,m,...}, and state Siis regarded as the status between epochs Ei-1and Ei. As LLP, performance, and random failure states are all considered,LLP replacement actions,performance recovery actions, and failure corrective actions should be determined on each decision epoch.

    LLP replacement actions denote maintenance actions of replacing an aero-engine LLP. When an LLP replacement action is performed, the definite LLP is replaced, and the LLP state is changed.LLP replacement actions are represented by {Arep,i|i=0,1,2,...}, where Arep,0denotes no LLP replaced and Arep,m(m≠0) denotes LLP m replaced. When Arep,mis performed, the LLP m state is changed to the all-new state.

    Performance recovery actions denote maintenance actions of recovering aero-engine performance. When a performance recovery action is performed, the performance state is recovered by a definite level, and the performance state is changed.Performance recovery actions are represented by{Arec,j|j=0,1,2,...}, where Arec,0denotes no performance recovery action performed and Arec,m(m≠0)denotes the action of recovering m performance levels.

    Failure corrective actions denote maintenance actions of making a failure-state aero-engine back to the running state.When the aero-engine is trapped in the random failure state,a failure corrective action should be performed.Failure corrective actions are represented by Acor={Acor,0,Acor,1}, where Acor,0denotes no corrective maintenance performed and Acor,1denotes corrective maintenance performed.

    From the above, aero-engine maintenance actions are represented by

    Because an aero-engine does not operate during the maintenance process, maintenance actions are assumed to be ‘‘instantaneous” in the proposed optimization algorithm.39

    3.3. State transition

    In the reinforcement learning framework,the aero-engine state is changed by performed maintenance actions.Thus,LLP,performance, and random failure state transitions are illustrated as follows.

    (1) LLP state transition

    In engineering,the LLP state is measured by the flight-cycle or flight-hour,which increases directly as an aero-engine operates. The LLP state would be recovered to the all-new state when an LLP replacement action is performed. Thus, the LLP state transfers directly, without uncertainty.

    When action Arep,0is performed,LLP state Tiwould transfer to the definite state Ti+1,that is,p(Ti+1|Ti,Arep,0)=1.When action Arep,1is performed, LLP state Tiwould transfer to T0,that is, p(T0|Ti,Arep,1)=1. A schematic diagram of LLP state transition is shown in Fig. 4.

    (2) Performance state transition

    Aero-engine performance deterioration is inevitable in engineering. As the imperfect maintenance concept and random factors are considered in the proposed optimization algorithm,the CBM strategy is formulated as a discrete MDP. Probability matrices are adopted in performance state transition.

    (

    Because the maintenance concept of ‘‘a(chǎn)s good as new” has been proven to be far from the truth,40a more realistic concept of imperfect repair is adopted for performance recovery actions.41,42That is,the performance state cannot be recovered to the all-new state by any maintenance actions, and the performance state would transfer according to the transition probability matrices. When action Arec,m(m >0) is performed,the performance state would transfer from Dito Di-maccording to the probability matrix [p(Di-m|Di,Arec,m,i-m >0)]. A schematic diagram of performance state transition is shown in Fig. 5.

    The performance state transition probability matrices can be calculated by survival analysis based on the Weibull distribution.43,44

    Fig. 4 Schematic diagram of LLP state transition.

    Fig. 5 Schematic diagram of performance state transition.

    (3) Random failure state transition

    As an aero-engine may be in the random fault state occasionally, the CM strategy is also formulated as a discrete MDP. The random failure state would transfer according to probability matrices.

    In the proposed optimization algorithm, an aero-engine may fall into the random failure state F1by probability p(F1|Dj,F(xiàn)0;Acor,0). To be more realistic, it is assumed that the random failure probability is lower when the aero-engine is in a better performance state. Thus, the random failure probability p(F1|Dj,F(xiàn)0) is related to the performance state Dj.When the aero-engine is in the random failure state,corrective maintenance should be performed to drive it to the working state. It is assumed that the corrective maintenance is completely efficient, and the transition probability of the corrective maintenance is represented as p(F0|Dj,F(xiàn)1;Acor,1)=1.A schematic diagram of random failure state transition is shown in Fig. 6.

    From the above, the state transition matrix on Eiis represented by p(Si+1|Si,Ai), where Si+1denotes the aero-engine state on Ei+1, Sidenotes the aero-engine state on Ei, and Aidenotes the performed maintenance action on Ei. Different sub-states would transfer according to different modes, as illustrated above.

    3.4. Total cost and optimization

    In reinforcement learning, an agent chooses maintenance actions according to action costs.44,45In the proposed optimization algorithm, the optimal objective is to obtain a maintenance policy whose total cost is the minimum in the long run.The maintenance cost on decision epoch Ekis calculated by

    where ckdenotes the maintenance cost on decision epoch Ek;Cope,kdenotes the operating cost; Crep,k,Crec,k, Ccor,k, and Cotherdenote the LLP replacement cost, the performance recovery cost,the corrective maintenance cost,and other costs,respectively.

    Fig. 6 Schematic diagram of random failure state transition.

    In general, a good-performance-state aero-engine would have a better economic efficiency and a lower random failure rate. Thus, Cope,kis determined by the performance state.When an LLP replacement action is performed, material and replacement costs are both counted in Crep,k(Arep,k). LLP replacement costs vary from different LLP replacement actions. The performance recovery cost is represented by Crec,k(Arec,k), and when m >n, Crec,i(Arec,m)>Crec,i(Arec,n).The corrective maintenance cost Ccor,kis counted when a corrective maintenance is performed.

    As the life-cycle maintenance policy is optimized by the proposed optimization algorithm, the future maintenance cost should be counted in optimization. In reinforcement learning,a discount factor is adopted to address long-term optimization.Thus,the optimal objective of the proposed optimization algorithm is denoted as

    where C denotes the discounted future cost, and γ(γ ∈[0,1])denotes the discount factor, representing the relative impact of future action costs.

    In reinforcement learning, when a larger γ is adopted,future action costs would leave a greater impact on maintenance action selection.That is,when γ=0,the optimized policy is shortsighted,and the maintenance action is chosen by the current cost; when γ=1, all the future actions are considered in action selection,which would bring a heavy calculation burden. Thus, a balance between future costs and the calculation burden should be determined. Thus, the discount factor γ should be set as γ ∈(0,1), for example, γ=0.9.

    As the Gauss-Seidel value iteration algorithm is an effective reinforcement learning algorithm, which is widely used in policy optimization, it is adopted to seek for the maintenance action collection whose discounted long-term cost is the minimum.

    4. Numerical experiments of maintenance policy optimization

    Two numerical experiments were used to illustrate the proposed aero-engine life-cycle maintenance policy optimization algorithm in detail. According to the reinforcement learning framework,determinations of aero-engine states,maintenance actions, state transition matrices, and total cost matrices are described firstly.As traditional methods are unable to address the long service-life and the hybrid maintenance strategy synchronously,they were not adopted as the benchmark methods.The reinforcement learning algorithm of Gauss-Seidel value iteration was adopted in the experiments.

    4.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state should be determined firstly. As the multi-dimensional state space was adopted to represent the aero-engine state, performance, LLP, and random failure states were all considered in the first numerical experiment.

    The EGTM was adopted to represent the aero-engine performance state. For convenience, the EGTM time series were divided into several levels. In a sample fleet, there were three performance recovery actions, including minimal repair,medium repair, and overhaul repair. According to the performance recovery assumptions in Section 4.2,there should be at least five performance levels to fully illustrate the three performance recovery actions. In reinforcement learning, as more performance levels would make the optimization algorithm more complicated, five levels were able to present the aeroengine performance state. In the numerical experiment, the performance state was divided into five levels, denoted as{D1,D2,D3,D4,D5}, from good to bad, where D5denoted the worst performance level. Besides, the all-new performance state was denoted as D0.

    Because performance and random failure states were both transferred by probabilities, the random failure state was regarded as a specific‘‘performance state”.Thus,in the numerical experiment, performance and random failure states were represented by one state-space dimension, denoted as{D0,D1,D2,D3,D4,D5,F(xiàn)}.

    Although there were several LLPs in an aero-engine, for convenience,LLPs with the same life limitation were regarded as the same LLP type. In the first numerical experiment, one LLP type was taken into account,and the LLP state was measured by flight-cycles. Referring to the adopted performance state levels,the LLP state was divided into five levels,denoted as {T1,T2,T3,T4,T5}. Besides, T0denoted the all-new state.Thus, the aero-engine state was represented by a twodimensional state space, denoted as

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state F; Tj(j=0,1,2,3,4,5)denoted the LLP state. The performance and random failure states were denoted as the first dimension; the LLP state was denoted as the second dimension.

    Fig. 7 Schematic diagrams of performance recovery action effects on performance states.

    4.2. Maintenance actions

    In reinforcement learning, the aero-engine state was changed by performed maintenance actions, and maintenance actions should be determined according to the aero-engine state.Firstly, according to traditional and practical matters,46two performance recovery assumptions were made in the numerical experiment as follows:

    (1) An aero-engine could not be recovered to the all-new performance state by any performance recovery actions.

    (2) No maintenance action should be performed when the aero-engine was in the worst performance state.

    According to the sample fleet operation, three actions of minimal repair, medium repair, and overhaul repair were included in performance recovery actions. Thus, three performance recovery actions were adopted in the numerical experiment, denoted as Arec={Arec,1,Arec,2,Arec,3}, where Arec-h1was defined as the action to recover the performance state from Dxto Dx-1, when x-1 >0, or to keep in the performance state Dx,when x=1;Arec,2was defined as the action to recover the performance state from Dxto Dx-2, when x-2 >0, or to recover the performance state from Dxto D1,when 1 ≤x ≤2;Arec,3was defined as the action to recover the performance state from Dxto Dx-3, when x-3 >0, or to recover the performance state from Dxto D1, when 1 ≤x ≤3. Besides, Arec,0denoted no performance recovery action was performed.In the reinforcement learning framework, Fig. 7 shows performance recovery action effects on performance states.

    In the numerical experiment, the random failure state was regarded as a specific ‘‘performance state”. Thus, it was assumed that performance recovery actions could drive the random failure aero-engine back to the working state.

    As one LLP type was considered,LLP replacement actions were denoted as Arep={Arep,0,Arep,1}. Fig. 8 shows the LLP replacement action effect on LLP states.

    From the above, maintenance actions in the numerical experiment were represented by

    4.3. State transition

    Fig. 8 Schematic diagram of the LLP replacement action effect on LLP states.

    Based on the aforementioned methods, performance state Diand transition probability matrix P0were obtained from the sample fleet, as shown in

    Transition matrices of Arec,1, Arec,2, and Arec,3were P1, P2,and P3, denoted as

    where nllpdenoted the number of LLP types, and nrecadenoted the number of performance recovery actions.

    4.4. Maintenance policy optimization

    Based on the aforementioned methods, aero-engine states,maintenance actions, and state transition matrices were all determined. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy. The flow diagram of the proposed aero-engine maintenance policy optimization algorithm is shown in Fig. 9.

    Table 2 Transition probability matrix of an LLP replacement action.

    Fig. 9 Flow diagram of maintenance policy optimization.

    Because real cost data was unavailable, a hypothetical cost matrix was adopted in the numerical experiment. In engineering,the cost matrix may change according to the actual maintenance cost, and it would not distort the analysis of simulation results. In the numerical experiment, the LLP replacement cost Cllpwas assumed to be 350, while the performance recovery costs Crep,1, Crep,2, and Crep,3were assumed to be 300, 350, and 500. As an aero-engine has long service-life,future maintenance actions should be fully considered in lifecycle maintenance policy optimization.Thus,a larger discount factor γ=0.9 was adopted.In contrast to the Jacobi value iteration algorithm, the Gauss-Seidel value iteration algorithm has a faster convergence ability. Thus, in the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy.

    As the aero-engine state was represented by a twodimensional space, the optimal maintenance policy was presented by a two-dimensional policy map, shown in Fig. 10.

    Optimal maintenance actions on each decision epoch were shown in the policy map,and decision epochs were represented by aero-engine states. In the maintenance policy map of Fig. 10, the LLP state was regarded as the ordinate, and the performance state was regarded as the abscissa. Different maintenance actions were presented in different colors and shapes. In the legend, A0denoted action { Arep,0,Arec,0} ; A1denoted action { Arep,0,Arec,1} ; A2denoted action{ Arep,0,Arec,2} ;A4denoted action{ Arep,1,Arec,0} ;A6denoted action { Arep,1,Arec,2} . In engineering, a maintenance policy could be obtained according to the aero-engine state.

    Fig. 10 Maintenance policy map of the numerical experiment.

    4.5. Algorithm performance analysis

    In this section,a more complex numerical experiment was conducted to illustrate the proposed optimization algorithm with more detailed instructions. In engineering, an aero-engine is usually composed of more than one LLP type. Some LLP states were measured by flight-hour, different from the one in the first numerical experiment. Thus, a more complex numerical experiment with two LLP types was conducted.To distinguish the two numerical experiments,the first numerical experiment was named as Experiment 1,and the numerical experiment in this section was named as Experiment 2.

    Table 3 Transition matrices of no LLP replacement action.

    Table 4 Transition probability matrices of LLP1 replacement actions.

    4.5.1. Aero-engine states

    Two LLP types were considered in Experiment 2,and the two LLP states were measured by different units: one was flightcycle and the other was flight-hour.Thus,the aero-engine state was denoted as a three-dimensional state space. The three dimensions included the performance state,the LLP state represented by flight-cycle(LLP1), and the LLP state represented by flight-hour (LLP2).

    Corresponding to Experiment 1, in addition to all-new states T0and L0, the LLP1 state was divided into two levels,denoted as {T0,T1,T2};the LLP2 state was divided into three levels, denoted as {L0,L1,L2,L3}. Same as in Experiment 1,the performance state was represented by five levels, and the random failure state was regarded as a specific ‘‘performance state”. Thus, the aero-engine state space was represented by

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state; Tjdenoted the LLP1 state;Lkdenoted the LLP2 state.

    4.5.2. Maintenance actions and state transition

    In Experiment 2, LLP1 and LLP2 replacement actions were denoted as ArepT={ArepT,0,ArepT,1} and ArepL={ArepL,0,ArepL,1}. Performance recovery actions were the same as those in Experiment 1. Performance recovery assumptions were also available.Maintenance actions in Experiment 2 were represented by

    Similar to LLP1, the LLP2 state would transfer as the flight-hour increasing, and transition probabilities were unavailable for LLP2. However, aero-engine state transition matrices were changed by LLP2.

    On Em, action Am={ Arec,i,ArepT,0,ArepL,0|i=0,1,2,3}denoted no LLP replaced, and transition matrices are presented in Table 3, denoted as p(S,Arec,i,ArepT,0,ArepL,0|i=0,1,2,3)= p3{PF,i,,PN,PR,i|i=0,1,2,3}. The concrete forms of PF,, and PRwere the same as those in Experiment 1.

    State transition matrices of Am={ Arec,i,ArepT,1,ArepL,0|i=0,1,2,3} are presented in Table 4, denoted as p(S,Arec,i,ArepT,1,ArepL,0|i = 0,1,2,3) = p4{ PF,i,,PN,PR,i|i = 0,1,2,3} .

    State transition matrices of Am={Arec,i,ArepT,0,ArepL,1|i=0,1,2,3} are presented in Table 5, denoted as p(S,Arec,i,ArepT,0,ArepL,1|i = 0,1,2,3) = p5{PF,i,,PN,PR,i|i = 0,1,2,3}.

    State transition matrices of Am={Arec,i,ArepT,1,ArepL,1|i=0,1,2,3} are presented in Table 6, denoted as p(S,Arec,i,ArepT,1,ArepL,1|i = 0,1,2,3) = p6{PF,i,,PN,PR,i|i = 0,1,2,3}.

    4.5.3. Maintenance policy optimization

    In Experiment 2, the reinforcement learning algorithm of Gauss-Seidel value iteration was also adopted to optimize the maintenance policy. Hypothetical costs were adopted,and the LLP1 replacement cost Cllp,1was assumed to be 300 while the LLP2 replacement cost Cllp,2was assumed to be 600. The three performance recovery action costs were assumed to be 200, 500, and 800, respectively. The discount factor was set as γ=0.9. Because the aero-engine state was represented by a three-dimensional space, including the LLP1, LLP2, and performance states, the optimal maintenance policy was represented by a three-dimensional policy map, shown in Fig. 11.

    Table 6 Transition probability matrices of LLP1 and LLP2 replacement actions.

    Fig. 11 Maintenance policy map of Experiment 2.

    Table 7 Algorithm information of two experiments.

    In the three-dimensional policy map,the LLP1,LLP2,and performance states were regarded as x axis, z axis, and y axis,respectively.The maintenance actions were presented in different colors and shapes.In the legend of the maintenance policy map in Fig. 11, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action{Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action {Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.Based on the policy map, a maintenance policy was obtained according to the aero-engine state.

    4.5.4. Algorithm analysis

    In aforementioned two numerical experiments, a twodimensional state space was adopted in Experiment 1, while a three-dimensional state space was adopted in Experiment 2.It was obvious that state transition matrices of Experiment 2 were more complicated. Thus, the algorithm would become complex as the state number increased. The algorithm information of the two numerical experiments is shown in Table 7.

    In the aforementioned numerical experiments,performance and random failure states were defined as probability states,because they transferred by probabilities. LLP states did not transfer by probabilities, and were defined as definite states.As shown in the two numerical experiments, state transition matrix forms were impacted by definite states. Transition probabilities were impacted by probability states.In reinforcement learning, a larger state space would make the algorithm more complicated, and more iterations were needed to seek for the optimal policy. The impact of aero-engine state space complexity on the algorithm was obvious.

    As the discount factor is an important coefficient in reinforcement learning, the discount factor impact was analyzed by contrast experiments, which were based on Experiment 1.In addition to the discount factor, the other parameters were all the same as those in Experiment 1. Maintenance policy maps of discount factor analysis are shown in subgraphs of Fig. 12.

    Fig. 12 Policy maps of discount factor analysis in Example 1.

    In the policy maps,different maintenance actions were represented by different colors and shapes. In the legend of Fig. 12, A0denoted action {Arep,0,Arec,0}; A1denoted action{Arep,0,Arec,1}; A2denoted action {Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A6denoted action {Arep,1,Arec,2}. The analysis experiments showed that when the discount factor was set as γ >0.43,the optimal policy maps were the same as those in Fig. 10;when the discount factor was set as γ <0.35, the optimal policy maps were the same as those in Fig. 12(c). In Fig. 12, discount factors were set as γ = 0.43, γ = 0.41, and γ = 0.35,respectively. The analysis experiments showed that when a smaller discount factor was adopted, more low-cost maintenance actions were adopted in the optimal maintenance policy.It was consistent with the aforementioned analysis of the discount factor.

    As hypothetical maintenance costs were adopted in numerical experiments, cost impacts were analyzed. Based on the aforementioned experiments, the medium performance recovery cost was set as Crep,2=350, and it was regarded as the benchmark cost. Maintenance policy maps of different cost ratios are shown in subgraphs of Figs. 13 and 14.

    Fig. 13 Policy maps of performance recovery cost analysis in Example 1.

    In the legend of Fig. 13, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A3denoted action {Arep,0,Arec,3}; A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A7denoted action {Arep,1,Arec,3}. In the legend of Fig. 14, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A3denoted action {Arec,3,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A8denoted action {Arec,3,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1};A14denoted action{Arec,3,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.

    Fig. 14 Policy maps of performance recovery cost analysis in Example 2.

    Fig. 15 Policy maps of LLP replacement cost analysis in Example 1.

    As shown in Figs.13(a)and 14(a),showed the optimal policy when Crec,1decreased; Figs. 13(b) and 14(b) showed the optimal policy when Crec,3decreased; Figs. 13(c) and 14(c)showed the optimal policy when Crec,1and Crec,3decreased simultaneously. As shown in Figs. 13(c) and 14(c), when Crec,1and Crec,3decreased simultaneously, optimal policy changes were not obvious.

    The impact of the LLP replacement cost on the optimal maintenance policy was analyzed by contrast experiments.Experiment results showed that the optimal maintenance policy would not change as the LLP replacement cost increasing.However,LLP replacement action times would increase as the LLP replacement cost decreasing.Optimal policy maps of LLP replacement cost analysis are shown in Figs. 15-17.

    In the legend of Fig. 15, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A6denoted action {Arep,1,Arec,2}. In the legends of Figs. 16 and 17, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A2denoted action {Arec,2,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A7denoted action {Arec,2,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A13denoted action {Arec,2,ArepT,1,ArepL,1}; A15denoted action{Arec,0,ArepT,1,ArepL,1}.

    As shown in Fig. 15, the LLP replacement cost decreased from subgraph (a) to (c). In Figs. 16 and 17, LLP1 and LLP2 replacement costs decreased from subgraph (a) to (b).It was shown that LLP replacement times would increase as the LLP replacement cost decreasing, and policy changes appeared on interface decision epochs.

    In the aforementioned experiments, the impact of the LLP residual life was not considered. Thus, based on the assumption that the random failure probability would increase as the LLP residual life decreasing, contrast experiments were performed to analyze the optimization algorithm.Optimal policy maps are shown in Fig. 18.

    Fig. 16 Policy maps of LLP1 replacement cost analysis in Example 2.

    Fig. 17 Policy maps of LLP2 replacement cost analysis in Example 2.

    In the legend of Fig. 18, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}. Based on Experiment 2, according to the residual life of the elder LLP, random failure probabilities were enhanced by 5%, 10%, and 15% respectively from subgraph (a) to (c)of Fig. 18. Policy maps showed no variation. Thus, the LLP residual life may not affect the optimal maintenance policy.

    5. Conclusions

    Based on the reinforcement learning approach,an aero-engine life-cycle maintenance policy optimization algorithm was proposed, which was able to address the long service-life and the hybrid maintenance strategy synchronously. To address the hybrid maintenance strategy,the multi-dimensional state space was adopted to represent the aero-engine state. Based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the life-cycle maintenance policy.

    Compared with traditional optimization methods, the optimal maintenance policy was used to indicate when and how to repair an aero-engine, taking the place of maintenance intervals and work-scopes in traditional methods.Meanwhile, the aero-engine long service-life, the hybrid maintenance strategy, and random factor destabilization were all addressed by the proposed optimization algorithm.Because few historical data was available for training the pre-specified optimization model of the aero-engine lifecycle maintenance policy, the reinforcement learning approach provided an appropriate way. In the reinforcement learning framework, the aero-engine state space, maintenance actions, and state transition matrices were determined according to aero-engine real-life operation. The Gauss-Seidel value iteration algorithm was employed to solve the long-term decision-making problem. The proposed optimization algorithm would help in making a wiser aero-engine life-cycle maintenance policy, resulting in a lower life-cycle maintenance cost. Two numerical experiments and algorithm analyses were employed to illustrate the proposed optimization algorithm in detail.

    As real aero-engine maintenance cost data was unavailable,hypothetical data was adopted in the numerical experiments.In future studies, maintenance cost calculation methods deserve further attention to improve the applicability of the proposed optimization algorithm.

    Fig. 18 Policy maps with LLP lifetime impact on transition probability.

    Acknowledgments

    The authors thank anonymous reviewers for their critical and constructive review of the manuscript. This work was cosupported by the Key National Natural Science Foundation of China (No. U1533202), the Civil Aviation Administration of China (No. MHRD20150104), and the Shandong Independent Innovation and Achievements Transformation Fund,China (No. 2014CGZH1101).

    亚洲五月色婷婷综合| 人人妻人人澡人人爽人人夜夜| 99国产综合亚洲精品| 久久性视频一级片| 欧美精品av麻豆av| 欧美不卡视频在线免费观看 | 久久天堂一区二区三区四区| 国产精品av久久久久免费| 久久午夜亚洲精品久久| av国产精品久久久久影院| 中文亚洲av片在线观看爽 | 亚洲精品久久午夜乱码| 亚洲精品国产区一区二| 99re6热这里在线精品视频| 丝袜人妻中文字幕| 精品第一国产精品| 精品一区二区三区视频在线观看免费 | 动漫黄色视频在线观看| 在线永久观看黄色视频| 最新的欧美精品一区二区| 人妻久久中文字幕网| 成人永久免费在线观看视频| 一级a爱片免费观看的视频| 国产精品.久久久| 午夜精品在线福利| 一级毛片精品| 国产成人欧美| 国产一区二区三区综合在线观看| 一进一出抽搐动态| 精品久久久精品久久久| 国产又色又爽无遮挡免费看| a级毛片在线看网站| 高清av免费在线| 亚洲va日本ⅴa欧美va伊人久久| 中文欧美无线码| 美女高潮到喷水免费观看| 丝袜美腿诱惑在线| 视频区欧美日本亚洲| 国产主播在线观看一区二区| 久久国产乱子伦精品免费另类| 国产欧美日韩一区二区三区在线| 亚洲三区欧美一区| 久久香蕉精品热| 日本黄色视频三级网站网址 | 亚洲熟妇熟女久久| 久久中文看片网| 国产av一区二区精品久久| 男人舔女人的私密视频| 国产精品亚洲一级av第二区| 亚洲中文字幕日韩| 人人妻人人爽人人添夜夜欢视频| 中文字幕人妻熟女乱码| 免费一级毛片在线播放高清视频 | 一边摸一边做爽爽视频免费| 人妻一区二区av| 国产精品久久久久久人妻精品电影| 热99re8久久精品国产| 日日摸夜夜添夜夜添小说| 老司机靠b影院| 亚洲熟妇中文字幕五十中出 | 精品国内亚洲2022精品成人 | 母亲3免费完整高清在线观看| 日本精品一区二区三区蜜桃| 午夜视频精品福利| videosex国产| 免费看十八禁软件| 久久午夜亚洲精品久久| 91字幕亚洲| 极品少妇高潮喷水抽搐| 动漫黄色视频在线观看| 日韩欧美国产一区二区入口| 久久久久久久国产电影| 动漫黄色视频在线观看| 悠悠久久av| 久久久久久久国产电影| videos熟女内射| 搡老岳熟女国产| 18禁国产床啪视频网站| 天天添夜夜摸| 人人妻人人澡人人爽人人夜夜| 视频区欧美日本亚洲| 老鸭窝网址在线观看| 久久香蕉激情| 久久草成人影院| 国产日韩一区二区三区精品不卡| 亚洲欧美一区二区三区久久| 精品福利观看| 丝瓜视频免费看黄片| 午夜福利欧美成人| 乱人伦中国视频| 午夜免费鲁丝| 两个人看的免费小视频| 热re99久久国产66热| 夜夜躁狠狠躁天天躁| www.熟女人妻精品国产| 两个人看的免费小视频| 黄色a级毛片大全视频| 日日夜夜操网爽| 91老司机精品| 窝窝影院91人妻| 搡老熟女国产l中国老女人| 99国产精品一区二区三区| 欧美性长视频在线观看| 欧美不卡视频在线免费观看 | 美女 人体艺术 gogo| 欧美最黄视频在线播放免费 | 久久天堂一区二区三区四区| 久久久精品免费免费高清| 99精品欧美一区二区三区四区| 女人久久www免费人成看片| 久久中文字幕一级| 日韩人妻精品一区2区三区| 午夜精品久久久久久毛片777| x7x7x7水蜜桃| 免费av中文字幕在线| 亚洲情色 制服丝袜| 亚洲成人手机| 激情在线观看视频在线高清 | 人妻 亚洲 视频| 国产乱人伦免费视频| 悠悠久久av| 久久久国产欧美日韩av| 少妇被粗大的猛进出69影院| 国产欧美亚洲国产| 久久中文字幕一级| 国产av又大| 99久久综合精品五月天人人| 伦理电影免费视频| 亚洲国产精品合色在线| 精品乱码久久久久久99久播| x7x7x7水蜜桃| 麻豆乱淫一区二区| 欧美激情高清一区二区三区| 高清在线国产一区| 999精品在线视频| 国产精品.久久久| 三级毛片av免费| 国产单亲对白刺激| 极品人妻少妇av视频| 精品久久蜜臀av无| 麻豆成人av在线观看| 天堂动漫精品| 欧美成人免费av一区二区三区 | 精品一品国产午夜福利视频| 嫁个100分男人电影在线观看| 丁香六月欧美| av欧美777| 国产色视频综合| 中文字幕高清在线视频| 国产精品久久久人人做人人爽| 国产成人精品无人区| 极品教师在线免费播放| ponron亚洲| 一级a爱片免费观看的视频| 免费少妇av软件| 精品人妻在线不人妻| 高清黄色对白视频在线免费看| 久久久国产成人精品二区 | 高清在线国产一区| 国产精品一区二区精品视频观看| 美女视频免费永久观看网站| 欧美精品高潮呻吟av久久| 久久久水蜜桃国产精品网| 男女免费视频国产| 国产精华一区二区三区| 亚洲一卡2卡3卡4卡5卡精品中文| 18禁裸乳无遮挡动漫免费视频| 欧美中文综合在线视频| 成人免费观看视频高清| 中亚洲国语对白在线视频| 亚洲av欧美aⅴ国产| 国产成人一区二区三区免费视频网站| 免费av中文字幕在线| 久久国产亚洲av麻豆专区| 少妇 在线观看| 91成人精品电影| 国产在线一区二区三区精| 波多野结衣av一区二区av| 国产精品自产拍在线观看55亚洲 | 亚洲精品在线美女| 亚洲av美国av| 日韩大码丰满熟妇| 伦理电影免费视频| 日本黄色视频三级网站网址 | 久久 成人 亚洲| 激情视频va一区二区三区| 少妇被粗大的猛进出69影院| 欧美日韩成人在线一区二区| 国产高清国产精品国产三级| 国产乱人伦免费视频| 精品一品国产午夜福利视频| 最近最新免费中文字幕在线| av网站在线播放免费| 欧美 亚洲 国产 日韩一| 国产在线观看jvid| 久久久久久久午夜电影 | 91av网站免费观看| 日韩大码丰满熟妇| 国产精品一区二区在线不卡| 侵犯人妻中文字幕一二三四区| 久久精品国产亚洲av香蕉五月 | 国产xxxxx性猛交| 亚洲人成电影观看| 精品久久久精品久久久| 国产欧美日韩精品亚洲av| 中文字幕色久视频| 亚洲 国产 在线| 国产精品久久久久久人妻精品电影| av国产精品久久久久影院| 国产又色又爽无遮挡免费看| 老司机午夜十八禁免费视频| 精品一区二区三区视频在线观看免费 | 亚洲少妇的诱惑av| 高清毛片免费观看视频网站 | 美国免费a级毛片| 80岁老熟妇乱子伦牲交| bbb黄色大片| 色老头精品视频在线观看| 亚洲精品美女久久久久99蜜臀| 亚洲成国产人片在线观看| 一区二区三区激情视频| 狠狠狠狠99中文字幕| 亚洲欧美日韩高清在线视频| 在线永久观看黄色视频| 中出人妻视频一区二区| 成人国产一区最新在线观看| 国产精品偷伦视频观看了| 亚洲一区二区三区不卡视频| 大码成人一级视频| www.精华液| 女人被躁到高潮嗷嗷叫费观| a在线观看视频网站| 午夜福利影视在线免费观看| 日韩熟女老妇一区二区性免费视频| 亚洲avbb在线观看| 黑人巨大精品欧美一区二区蜜桃| 国产精品偷伦视频观看了| 欧美 亚洲 国产 日韩一| 天堂√8在线中文| 中出人妻视频一区二区| 99久久人妻综合| 视频在线观看一区二区三区| 国产亚洲欧美精品永久| 精品久久久久久久久久免费视频 | 国产亚洲精品第一综合不卡| 在线免费观看的www视频| 成在线人永久免费视频| av福利片在线| 精品少妇久久久久久888优播| 亚洲精品在线美女| 亚洲,欧美精品.| 黄片大片在线免费观看| 最新美女视频免费是黄的| 亚洲五月色婷婷综合| 亚洲成人国产一区在线观看| 久久精品91无色码中文字幕| 多毛熟女@视频| 一二三四在线观看免费中文在| 久久精品国产清高在天天线| 两性夫妻黄色片| 看片在线看免费视频| 亚洲专区中文字幕在线| 男女下面插进去视频免费观看| 免费一级毛片在线播放高清视频 | 成人免费观看视频高清| 午夜福利免费观看在线| 两个人看的免费小视频| 久久天堂一区二区三区四区| 夫妻午夜视频| av中文乱码字幕在线| 中文欧美无线码| 欧美日韩av久久| 久久人妻福利社区极品人妻图片| 国产成人精品久久二区二区免费| 亚洲精品美女久久av网站| √禁漫天堂资源中文www| 日韩免费av在线播放| 免费看十八禁软件| 又黄又爽又免费观看的视频| 国产有黄有色有爽视频| 9191精品国产免费久久| 欧美黑人欧美精品刺激| 制服人妻中文乱码| 国产精品久久久人人做人人爽| 欧美黄色淫秽网站| 亚洲全国av大片| 十八禁人妻一区二区| 亚洲自偷自拍图片 自拍| 日韩三级视频一区二区三区| 久久久久久久久免费视频了| 另类亚洲欧美激情| 精品欧美一区二区三区在线| 婷婷精品国产亚洲av在线 | 国产精品影院久久| 国产欧美日韩综合在线一区二区| 热re99久久精品国产66热6| cao死你这个sao货| 女性被躁到高潮视频| 亚洲性夜色夜夜综合| 亚洲色图综合在线观看| 久久精品国产清高在天天线| 正在播放国产对白刺激| 精品人妻在线不人妻| 中文字幕色久视频| 国产在线观看jvid| 国产精品亚洲av一区麻豆| 纯流量卡能插随身wifi吗| 99riav亚洲国产免费| 午夜福利影视在线免费观看| 在线永久观看黄色视频| 高清欧美精品videossex| 男女午夜视频在线观看| 久久精品人人爽人人爽视色| 午夜成年电影在线免费观看| 九色亚洲精品在线播放| 久久精品人人爽人人爽视色| 欧洲精品卡2卡3卡4卡5卡区| 亚洲熟女毛片儿| 久久国产精品大桥未久av| 久久人人爽av亚洲精品天堂| 日本五十路高清| 男女之事视频高清在线观看| 国产精品久久电影中文字幕 | 又黄又爽又免费观看的视频| 亚洲成人国产一区在线观看| 午夜老司机福利片| 国产精品久久久久成人av| 久久国产精品男人的天堂亚洲| 欧美亚洲 丝袜 人妻 在线| 老司机午夜十八禁免费视频| 在线观看舔阴道视频| 好男人电影高清在线观看| 国产又爽黄色视频| 久久九九热精品免费| 韩国av一区二区三区四区| 亚洲国产看品久久| 成人手机av| 亚洲自偷自拍图片 自拍| 深夜精品福利| 国产成人免费观看mmmm| 精品少妇一区二区三区视频日本电影| a级毛片在线看网站| 激情在线观看视频在线高清 | 精品一区二区三区视频在线观看免费 | 99精国产麻豆久久婷婷| 久久国产亚洲av麻豆专区| 国产欧美日韩精品亚洲av| 久久热在线av| 天堂动漫精品| 亚洲在线自拍视频| 十八禁网站免费在线| 夜夜躁狠狠躁天天躁| 99精品欧美一区二区三区四区| 最近最新中文字幕大全电影3 | 国产成人啪精品午夜网站| 一a级毛片在线观看| 成熟少妇高潮喷水视频| 精品人妻在线不人妻| 午夜成年电影在线免费观看| 他把我摸到了高潮在线观看| 国产在线精品亚洲第一网站| 亚洲精品美女久久久久99蜜臀| 黄色a级毛片大全视频| 一本一本久久a久久精品综合妖精| av天堂久久9| 亚洲第一青青草原| 亚洲情色 制服丝袜| 亚洲男人天堂网一区| 久久午夜综合久久蜜桃| 亚洲av成人av| 看免费av毛片| 美女高潮到喷水免费观看| 久久 成人 亚洲| 欧美日韩中文字幕国产精品一区二区三区 | 日韩欧美免费精品| 色老头精品视频在线观看| 男女下面插进去视频免费观看| 美女国产高潮福利片在线看| 电影成人av| 变态另类成人亚洲欧美熟女 | 欧美精品啪啪一区二区三区| 中文字幕av电影在线播放| 韩国精品一区二区三区| 黄频高清免费视频| 国产精品电影一区二区三区 | 麻豆国产av国片精品| 久久精品亚洲熟妇少妇任你| 18禁国产床啪视频网站| 国产成人欧美在线观看 | 亚洲熟女精品中文字幕| 欧美激情久久久久久爽电影 | 亚洲欧美激情在线| 久久中文字幕人妻熟女| 黄片播放在线免费| 国产又色又爽无遮挡免费看| 日本撒尿小便嘘嘘汇集6| 欧美亚洲日本最大视频资源| 国产精品久久视频播放| 欧美乱色亚洲激情| 欧美日本中文国产一区发布| 在线观看免费高清a一片| 欧美人与性动交α欧美精品济南到| 搡老乐熟女国产| 亚洲精品一二三| 国产aⅴ精品一区二区三区波| 亚洲精品一二三| 国产成人av激情在线播放| 99在线人妻在线中文字幕 | 狠狠婷婷综合久久久久久88av| 中文字幕精品免费在线观看视频| 久久久久久久久久久久大奶| 在线观看www视频免费| 欧美人与性动交α欧美精品济南到| 18禁观看日本| 久久久久视频综合| 国产精品久久久人人做人人爽| 久久热在线av| 日韩三级视频一区二区三区| 天天躁日日躁夜夜躁夜夜| 黄频高清免费视频| 亚洲欧美精品综合一区二区三区| 成年人黄色毛片网站| 精品电影一区二区在线| 久久久久久亚洲精品国产蜜桃av| 在线观看免费视频网站a站| 多毛熟女@视频| 免费在线观看完整版高清| 亚洲色图综合在线观看| 极品教师在线免费播放| 午夜激情av网站| 一边摸一边抽搐一进一小说 | 国产成人免费观看mmmm| 一区福利在线观看| 黄色a级毛片大全视频| 成人免费观看视频高清| 热99国产精品久久久久久7| 日韩人妻精品一区2区三区| 久久久精品区二区三区| 国产欧美日韩综合在线一区二区| 新久久久久国产一级毛片| 欧美成人免费av一区二区三区 | 成人三级做爰电影| 在线观看免费视频网站a站| 久久人妻av系列| 捣出白浆h1v1| 99久久99久久久精品蜜桃| 午夜福利在线免费观看网站| 曰老女人黄片| 久久精品91无色码中文字幕| 老汉色∧v一级毛片| 久久这里只有精品19| 一级a爱片免费观看的视频| 国内久久婷婷六月综合欲色啪| 国产精品av久久久久免费| 亚洲九九香蕉| 久久 成人 亚洲| 亚洲熟妇熟女久久| 国产精品九九99| 午夜福利免费观看在线| 亚洲欧美日韩高清在线视频| 亚洲精品一二三| 国产精品永久免费网站| 国产不卡一卡二| 免费不卡黄色视频| 国产精品综合久久久久久久免费 | 国内久久婷婷六月综合欲色啪| 亚洲欧美色中文字幕在线| 日韩一卡2卡3卡4卡2021年| 999久久久国产精品视频| 如日韩欧美国产精品一区二区三区| 国产亚洲av高清不卡| 夜夜爽天天搞| 免费在线观看视频国产中文字幕亚洲| 久久国产精品大桥未久av| 窝窝影院91人妻| 国产欧美日韩精品亚洲av| 国产aⅴ精品一区二区三区波| 久久国产精品男人的天堂亚洲| 免费在线观看亚洲国产| 亚洲伊人色综图| 性少妇av在线| 欧美+亚洲+日韩+国产| 欧美日韩视频精品一区| www.自偷自拍.com| 大片电影免费在线观看免费| 亚洲av成人av| 亚洲va日本ⅴa欧美va伊人久久| 亚洲人成电影观看| www.自偷自拍.com| 多毛熟女@视频| 亚洲av成人av| 在线十欧美十亚洲十日本专区| 大香蕉久久成人网| 男人舔女人的私密视频| 精品午夜福利视频在线观看一区| 亚洲男人天堂网一区| 亚洲午夜理论影院| 色94色欧美一区二区| 国产免费男女视频| 超色免费av| av天堂在线播放| 夫妻午夜视频| 视频区欧美日本亚洲| 欧美日韩精品网址| 亚洲一卡2卡3卡4卡5卡精品中文| 淫妇啪啪啪对白视频| 久久精品亚洲精品国产色婷小说| 日韩大码丰满熟妇| 日韩制服丝袜自拍偷拍| 国产精品欧美亚洲77777| 大香蕉久久网| 欧美日韩av久久| 在线观看日韩欧美| 啦啦啦免费观看视频1| 50天的宝宝边吃奶边哭怎么回事| 丝袜美腿诱惑在线| 日韩中文字幕欧美一区二区| 99久久精品国产亚洲精品| 国产蜜桃级精品一区二区三区 | 叶爱在线成人免费视频播放| 无限看片的www在线观看| 欧美激情高清一区二区三区| 99精品在免费线老司机午夜| 亚洲成人手机| 一二三四社区在线视频社区8| 伦理电影免费视频| a级片在线免费高清观看视频| 黑人猛操日本美女一级片| 日韩欧美一区二区三区在线观看 | 亚洲欧美一区二区三区久久| 中文字幕人妻丝袜制服| 不卡av一区二区三区| 交换朋友夫妻互换小说| 下体分泌物呈黄色| 18在线观看网站| 久久久久久免费高清国产稀缺| 两性夫妻黄色片| 亚洲 国产 在线| 女人被狂操c到高潮| 一级a爱片免费观看的视频| svipshipincom国产片| 黄色女人牲交| 黄色丝袜av网址大全| 美女高潮喷水抽搐中文字幕| 精品国内亚洲2022精品成人 | 女人精品久久久久毛片| 亚洲国产看品久久| 女人精品久久久久毛片| 免费少妇av软件| 777米奇影视久久| 久久精品国产亚洲av香蕉五月 | 国产主播在线观看一区二区| 国产高清视频在线播放一区| 麻豆国产av国片精品| 在线观看www视频免费| tocl精华| 久久人人97超碰香蕉20202| 中文字幕人妻熟女乱码| 黑人巨大精品欧美一区二区蜜桃| 精品一区二区三区四区五区乱码| 日韩免费av在线播放| 两人在一起打扑克的视频| 午夜免费成人在线视频| 欧美一级毛片孕妇| 久久精品国产亚洲av香蕉五月 | 两个人免费观看高清视频| 正在播放国产对白刺激| 深夜精品福利| 国产成人免费无遮挡视频| 精品国产一区二区三区久久久樱花| 在线十欧美十亚洲十日本专区| 国产精品久久视频播放| 欧美日韩黄片免| 国产精品秋霞免费鲁丝片| tube8黄色片| 国产精品欧美亚洲77777| 99精品欧美一区二区三区四区| 免费一级毛片在线播放高清视频 | 极品少妇高潮喷水抽搐| 精品久久久久久久久久免费视频 | 一区在线观看完整版| 国产高清激情床上av| 女人爽到高潮嗷嗷叫在线视频| 香蕉国产在线看| 精品一品国产午夜福利视频| 91在线观看av| 波多野结衣一区麻豆| 亚洲片人在线观看| 天天躁日日躁夜夜躁夜夜| 波多野结衣一区麻豆| 两个人免费观看高清视频| 国产亚洲精品一区二区www | 免费少妇av软件| 91精品国产国语对白视频| 欧美亚洲 丝袜 人妻 在线| 欧美在线黄色| 王馨瑶露胸无遮挡在线观看| 欧美成狂野欧美在线观看| 亚洲精品久久成人aⅴ小说| 精品卡一卡二卡四卡免费| tocl精华| 国产免费av片在线观看野外av| 国产免费男女视频| 女警被强在线播放| 免费在线观看完整版高清| 曰老女人黄片| 国产区一区二久久| 久久香蕉国产精品| 黄网站色视频无遮挡免费观看| 成人三级做爰电影| 国产精品自产拍在线观看55亚洲 | 中文字幕色久视频| 成人免费观看视频高清| 精品一区二区三区视频在线观看免费 | 69精品国产乱码久久久|