• <tr id="yyy80"></tr>
  • <sup id="yyy80"></sup>
  • <tfoot id="yyy80"><noscript id="yyy80"></noscript></tfoot>
  • 99热精品在线国产_美女午夜性视频免费_国产精品国产高清国产av_av欧美777_自拍偷自拍亚洲精品老妇_亚洲熟女精品中文字幕_www日本黄色视频网_国产精品野战在线观看 ?

    An aero-engine life-cycle maintenance policy optimization algorithm: Reinforcement learning approach

    2019-10-26 01:12:16ZhenLIShishengZHONGLinLIN
    CHINESE JOURNAL OF AERONAUTICS 2019年9期

    Zhen LI, Shisheng ZHONG, Lin LIN

    School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China

    KEYWORDS Aero-engine;Hybrid strategy;Maintenance policy;Optimization algorithm;Reinforcement learning

    Abstract An aero-engine maintenance policy plays a crucial role in reasonably reducing maintenance cost. An aero-engine is a type of complex equipment with long service-life. In engineering,a hybrid maintenance strategy is adopted to improve the aero-engine operational reliability. Thus,the long service-life and the hybrid maintenance strategy should be considered synchronously in aero-engine maintenance policy optimization. This paper proposes an aero-engine life-cycle maintenance policy optimization algorithm that synchronously considers the long service-life and the hybrid maintenance strategy. The reinforcement learning approach was adopted to illustrate the optimization framework, in which maintenance policy optimization was formulated as a Markov decision process. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the maintenance policy. Compared with traditional aero-engine maintenance policy optimization methods,the long service-life and the hybrid maintenance strategy could be addressed synchronously by the proposed algorithm. Two numerical experiments and algorithm analyses were performed to illustrate the optimization algorithm in detail.

    1. Introduction

    An aero-engine, which is composed of mechanic-electrichydraulic coupling systems,is the power plant of an aircraft.1,2It has been reported that more than 30% of aircraft mechanical problems are related to aero-engines, and the aero-engine maintenance cost contributes to about 30% of an airline's direct operating cost.3A maintenance optimization method provides an available way to reduce the maintenance cost reasonably.4In general, excessive maintenance is costly, while insufficient maintenance may lead to disasters.Thus,a maintenance policy plays a crucial role in balancing the maintenance cost and operational reliability.5However, it is not an easy work to optimize an aero-engine maintenance policy manually,especially taking the long service-life and the hybrid maintenance strategy into consideration synchronously.

    In engineering,a hybrid maintenance strategy is adopted to improve the civil aero-engine operational reliability.Strategies of Condition-Based Maintenance (CBM), Hard-time Maintenance (HM), and failure Corrective Maintenance (CM) are included in the hybrid maintenance strategy.6As aero-engine performance deterioration is inevitable,7gas path performance parameters are monitored for CBM. The Life Limit Part(LLP) should be replaced before its life limitation,8and HM is adopted. Moreover, CM is performed when an aeroengine is in the random failure state.Thus,the hybrid maintenance strategy should be considered in aero-engine maintenance policy optimization. However, few existing aero-engine maintenance optimization methods are able to address the hybrid strategy. At the same time, an aero-engine is a type of equipment with long service-life,9which should be synchronously considered in maintenance policy optimization.To the best of our knowledge, this is the first paper to investigate an aero-engine maintenance policy optimization method that can synchronously address the hybrid maintenance strategy and long service-life.

    In traditional aero-engine maintenance optimization methods, maintenance intervals are used to decide when to repair an aero-engine, and maintenance work-scopes indicate how to carry out maintenance actions. Maintenance intervals and work-scopes can be obtained by traditional separate models.For example, LLP replacement intervals and performance recovery intervals can be optimized separately by current models.10-12Based on maintenance intervals, maintenance workscopes are obtained by traditional decision-making models.13,14Based on traditional optimization methods, an aeroengine maintenance decision support system was proposed by Fu et al.,15in which maintenance interval and work-scope optimization models were presented. An optimization method for reliability-centered maintenance was proposed by Crocker and Kumar,16in which the concepts of soft life and hard life were used to optimize a military aero-engine maintenance policy.A multi-objective evolutionary algorithm was also adopted to solve the aero-engine maintenance scheduling problem,17taking the module exchange into consideration. To optimize an aero-engine maintenance policy, traditional optimization methods would become extremely complicated when the long service-life and the hybrid maintenance strategy are considered synchronously.18

    In general, machine learning methods include supervised learning, unsupervised learning, and reinforcement learning,19and the reinforcement learning method has attracted increasing interests in solving decision-making problems.20Reinforcement learning represents a machine learning method in which an agent learns how to behave through action rewards.Different from widely used supervised learning methods, there is no presentation of input and output pairs in reinforcement learning. In reinforcement learning, an agent chooses an available action according to the environment on each decision epoch.The chosen action changes the environment, along with a reward to the agent. The objective of the agent is to find the action collection, whose reward is maximal in the long run.

    Reinforcement learning methods have been adopted in energy system charging policy optimization,21,22energy system and distributed system schedule determination,23,24multiple robotic task optimization,25demand response optimization,26,27robust control optimization,28multiple satellites task planning,29et al. Although reinforcement learning methods have been successfully applied, they have not aroused much attention in aero-engine maintenance policy optimization. Reinforcement learning does provide a more appropriate way for aero-engine life-cycle maintenance policy optimization. Thus, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on reinforcement learning. The main contributions of this paper are as follows:

    (1) An aero-engine life-cycle maintenance policy optimization algorithm is proposed, which can synchronously address the aero-engine long service-life and the hybrid maintenance strategy.

    (2) To address the hybrid strategy, the aero-engine state is represented by a multi-dimensional state space.

    (3) Reinforcement learning is adopted to illustrate the maintenance policy optimization. In the reinforcement learning framework, maintenance policy optimization is formulated as a discrete Markov Decision Process(MDP), and the Gauss-Seidel value iteration algorithm is adopted to optimize the maintenance policy.

    The remainder of this paper is organized as follows.Section 2 introduces aero-engine maintenance policy optimization and the reinforcement learning approach, and the deficiencies of traditional optimization methods are analyzed.In Section 3, the proposed aero-engine life-cycle maintenance policy optimization algorithm is described in detail. In Section 4, two simulation experiments and algorithm analysis are used to illustrate the proposed optimization algorithm.Conclusions and future work are discussed in Section 5.

    2. Aero-engine maintenance policy optimization and reinforcement learning

    2.1. Aero-engine maintenance policy optimization

    A maintenance policy indicates when and how to repair an aero-engine. Traditionally, maintenance intervals indicate when to repair an aero-engine, which are obtained by traditional separate optimization models, addressing CBM, HM and,CM strategies separately.In traditional methods,maintenance work-scopes indicate how to repair an aero-engine,which are optimized based on the outputs of interval optimization models. In summary, there are several deficiencies in traditional aero-engine maintenance optimization methods, as follows:

    (1) In traditional methods,maintenance intervals and workscopes are optimized by separate models, and workscopes are optimized based on interval optimization results. Thus, the interactions of maintenance intervals and work-scopes are neglected.Moreover,interval optimization errors would be propagated to work-scope optimization.

    (2) Because the hybrid maintenance strategy is adopted for civil aero-engines, hybrid strategies should be addressed synchronously in optimization. However, traditional optimization methods address hybrid strategies separately, and interactions of hybrid strategies are neglected.

    (3) It is difficult for traditional optimization methods to address the aero-engine long-service life and the hybrid maintenance strategy synchronously.

    (4) Definite optimization results are obtained by traditional optimization methods. Due to random factors, definite optimization results may be poorly applicable in engineering.

    To deal with the aforementioned deficiencies of traditional methods, an aero-engine life-cycle maintenance policy optimization algorithm is proposed based on the reinforcement learning approach. Taking the place of maintenance intervals and work-scopes in traditional methods, the maintenance policy indicates when and how to repair an aero-engine.The proposed optimization algorithm is able to address the hybrid maintenance strategy and long service-life synchronously.

    To address the hybrid maintenance strategy, a multidimensional space is adopted to represent the aero-engine state.The maintenance strategies of CBM and CM are formulated as an MDP,and imperfect repair and random factors are all considered in state transition. Aero-engine maintenance policy optimization is illustrated by the reinforcement learning approach, and the Gauss-Seidel value iteration algorithm is adopted to seek for the optimal maintenance policy. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm makes it available to optimize the lifecycle maintenance policy. A comparison between traditional optimization methods and the proposed optimization algorithm is shown in Fig. 1.

    In Fig.1,traditional aero-engine maintenance optimization methods are shown on the left.Traditional methods are constituted by separate models,including an LLP interval optimization model, a performance interval optimization model, and a work-scope optimization model.LLP replacement and performance recovery intervals are obtained by separate optimization models. Based on the LLP replacement, performance recovery, and corrective maintenance intervals, the maintenance work-scope is optimized. The proposed aero-engine life-cycle maintenance policy optimization algorithm is shown on the right in Fig. 1. Based on the reinforcement learning framework, the strategies of HM, CM, and CBM are addressed synchronously by the proposed optimization algorithm. The traditional separate optimization models of the LLP replacement interval, the performance recovery interval,and the maintenance work-scope are replaced by the proposed optimization algorithm.

    2.2. Reinforcement learning approach

    Reinforcement learning is a machine learning method,which is widely used to solve multi-step, sequential-decision problems.Different from supervised learning, no pre-specified model is required in reinforcement learning. In aero-engine life-cycle maintenance policy optimization, few historical data is available for training the pre-specified mode. Thus, reinforcement learning provides a more appropriate way for aero-engine life-cycle maintenance policy optimization. Meanwhile, the aero-engine long service-life and the hybrid maintenance strategy can be addressed synchronously by reinforcement learning.

    In reinforcement learning, an agent takes on the work of optimizing an aero-engine maintenance policy. The agent is able to respond to dynamically changing aero-engine states through ongoing learning methods.30In aero-engine maintenance policy optimization, aero-engine states are represented by a multi-dimensional state space. To optimize the maintenance policy, the agent chooses a maintenance action according to the aero-engine state. The aero-engine state is changed by the chosen maintenance action,along with the maintenance cost, as shown in Fig. 2. The optimal objective of the agent is to find the maintenance action collection, whose total cost is the minimum in the long run.

    Fig. 2 Schematic diagram of reinforcement learning.

    Value iteration is a reinforcement learning algorithm that is widely adopted in solving decision-making problems. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm provides an appropriate way to optimize an aero-engine life-cycle maintenance policy. According to the Gauss-Seidel value iteration algorithm, an agent would run multiple episodes for the purpose of exploring and finding the optimal policy.The learning process is conducted for a sufficient number of iterations,and the total cost of each iteration is recorded.The minimum total cost is represented as Q-value,which is updated every iteration, and the Bellman equation is adopted as the updating mechanism in the Gauss-Seidel value iteration algorithm. The convergence of value iterative methods has been widely proven.31Thus, based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm.

    3. Aero-engine life-cycle maintenance policy optimization algorithm

    The reinforcement learning approach is adopted in the proposed aero-engine life-cycle maintenance policy optimization algorithm, in which the hybrid maintenance strategy and long service-life are considered synchronously.In the reinforcement learning framework, the aero-engine state, maintenance actions, state transition matrices, and maintenance costs should be determined. To address the hybrid maintenance strategy, a multi-dimensional state space is adopted to represent the aero-engine state, taking performance, LLP, and random failure states into consideration synchronously. The optimal objective of the proposed optimization algorithm is to obtain a maintenance policy whose total cost is the minimum in the long run. The reinforcement learning algorithm of Gauss-Seidel value iteration is adopted, addressing longterm optimization. In this section, the proposed optimization algorithm is described in detail.sub-state Yjdenotes the performance state, and Zkdenotes the random failure state.

    (1) LLP state

    In engineering,an LLP must be replaced before its life limitation,and the HM strategy is adopted for LLP maintenance.Traditionally, the flight-cycle or flight-hour is adopted to represent the LLP state. For example, a fan shaft is an LLP of a CFM56-5B aero-engine, and its life limitation is 30,000 flightcycles. That is, a fan shaft must be replaced before 30,000 flight-cycles. For convenience, the LLP state is represented by discrete time increments.32In the proposed optimization algorithm, the LLP state is represented by several state levels,denoted as {Ti|i=0,1,2,...}, where T0denotes the all-new state; when m <n, state Tnis ‘‘older” than state Tm. It is defined that an LLP is in state Tn, when tn-1<tllp≤tn; tn-1and tnare boundary values; tllpis an LLP real state, measured by flight-cycle or flight-hour.

    (2) Performance state

    As the CBM strategy is adopted in aero-engine maintenance, the aero-engine performance state should also be considered in the proposed optimization algorithm. Most of the in-service aero-engines are equipped with condition monitoring systems, and performance parameters are sent to ground by the aircraft communication addressing and reporting system in quasi real-time. Traditionally, the aero-engine performance state is assessed by performance parameters.33,34In engineering, the Exhaust Gas Temperature Margin (EGTM)is adopted as an aero-engine performance indicator.35The EGTM is defined as the margin temperature between exhaust gas temperature and red-line temperature. The aero-engine performance degrades as it operates, presented as EGTM declining.36,37When the EGTM is close to the limitation, performance recovery maintenance should be performed. The 1000 flight-cycles EGTM parameters of a CFM56-5B aeroengine are shown in Fig. 3.

    As EGTM parameters are typical time series,38it would make the algorithm extremely complicated when EGTM time series are adopted directly.For convenience,the CBM strategy is formulated as a discrete MDP in the reinforcement learning framework. Thus, the aero-engine performance state is represented by several levels, denoted as {Di|i=0,1,2,...}, where D0denotes the all-new performance state; when m <n, state Dnis worse than state Dm. It is defined that the performance state is Dn, when dn-1<dper≤dn, in which dn-1and dnare

    3.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state is changed by performed actions. As the hybrid maintenance strategy is considered in the proposed optimization algorithm,a multi-dimensional state space is adopted to represent the aero-engine state, in which performance, LLP, and random failure states are considered synchronously. The multidimensional state space is represented by S={Xi,Yj,Zk,...},where Xi,Yj,Zk,... denote the sub-states in the multidimensional state space. Each sub-state denotes one considered factor, for example, sub-state Xidenotes the LLP state,the boundary values;dperis the EGTM real value,measured by centigrade. Thus, in the proposed optimization algorithm, the aero-engine performance state is indicated by the performance levels.

    Fig. 3 1000 flight-cycles EGTM of a CFM56-5B aero-engine.

    (3) Random failure state

    Although an aero-engine is with high reliability, it is subjected to random failures in practice. When an aero-engine is in the random failure state, CM should be performed to drive it back to the working state. Unlike the LLP or performance state, the random failure state is represented by two levels,denoted as {F0,F(xiàn)1}, where F0denotes the working state and F1denotes the random failure state.

    From the above,when LLP,performance,and random failure states are all considered, the aero-engine state is represented by a multi-dimensional state space, denoted as

    where Tidenotes the LLP state, Djdenotes the performance state, and Fkdenotes the random failure state.

    3.2. Maintenance actions

    In reinforcement learning, maintenance actions are performed on every decision epoch. The decision epoch is denoted as{Eii=| 0,1,2,...,m,...}, and state Siis regarded as the status between epochs Ei-1and Ei. As LLP, performance, and random failure states are all considered,LLP replacement actions,performance recovery actions, and failure corrective actions should be determined on each decision epoch.

    LLP replacement actions denote maintenance actions of replacing an aero-engine LLP. When an LLP replacement action is performed, the definite LLP is replaced, and the LLP state is changed.LLP replacement actions are represented by {Arep,i|i=0,1,2,...}, where Arep,0denotes no LLP replaced and Arep,m(m≠0) denotes LLP m replaced. When Arep,mis performed, the LLP m state is changed to the all-new state.

    Performance recovery actions denote maintenance actions of recovering aero-engine performance. When a performance recovery action is performed, the performance state is recovered by a definite level, and the performance state is changed.Performance recovery actions are represented by{Arec,j|j=0,1,2,...}, where Arec,0denotes no performance recovery action performed and Arec,m(m≠0)denotes the action of recovering m performance levels.

    Failure corrective actions denote maintenance actions of making a failure-state aero-engine back to the running state.When the aero-engine is trapped in the random failure state,a failure corrective action should be performed.Failure corrective actions are represented by Acor={Acor,0,Acor,1}, where Acor,0denotes no corrective maintenance performed and Acor,1denotes corrective maintenance performed.

    From the above, aero-engine maintenance actions are represented by

    Because an aero-engine does not operate during the maintenance process, maintenance actions are assumed to be ‘‘instantaneous” in the proposed optimization algorithm.39

    3.3. State transition

    In the reinforcement learning framework,the aero-engine state is changed by performed maintenance actions.Thus,LLP,performance, and random failure state transitions are illustrated as follows.

    (1) LLP state transition

    In engineering,the LLP state is measured by the flight-cycle or flight-hour,which increases directly as an aero-engine operates. The LLP state would be recovered to the all-new state when an LLP replacement action is performed. Thus, the LLP state transfers directly, without uncertainty.

    When action Arep,0is performed,LLP state Tiwould transfer to the definite state Ti+1,that is,p(Ti+1|Ti,Arep,0)=1.When action Arep,1is performed, LLP state Tiwould transfer to T0,that is, p(T0|Ti,Arep,1)=1. A schematic diagram of LLP state transition is shown in Fig. 4.

    (2) Performance state transition

    Aero-engine performance deterioration is inevitable in engineering. As the imperfect maintenance concept and random factors are considered in the proposed optimization algorithm,the CBM strategy is formulated as a discrete MDP. Probability matrices are adopted in performance state transition.

    (

    Because the maintenance concept of ‘‘a(chǎn)s good as new” has been proven to be far from the truth,40a more realistic concept of imperfect repair is adopted for performance recovery actions.41,42That is,the performance state cannot be recovered to the all-new state by any maintenance actions, and the performance state would transfer according to the transition probability matrices. When action Arec,m(m >0) is performed,the performance state would transfer from Dito Di-maccording to the probability matrix [p(Di-m|Di,Arec,m,i-m >0)]. A schematic diagram of performance state transition is shown in Fig. 5.

    The performance state transition probability matrices can be calculated by survival analysis based on the Weibull distribution.43,44

    Fig. 4 Schematic diagram of LLP state transition.

    Fig. 5 Schematic diagram of performance state transition.

    (3) Random failure state transition

    As an aero-engine may be in the random fault state occasionally, the CM strategy is also formulated as a discrete MDP. The random failure state would transfer according to probability matrices.

    In the proposed optimization algorithm, an aero-engine may fall into the random failure state F1by probability p(F1|Dj,F(xiàn)0;Acor,0). To be more realistic, it is assumed that the random failure probability is lower when the aero-engine is in a better performance state. Thus, the random failure probability p(F1|Dj,F(xiàn)0) is related to the performance state Dj.When the aero-engine is in the random failure state,corrective maintenance should be performed to drive it to the working state. It is assumed that the corrective maintenance is completely efficient, and the transition probability of the corrective maintenance is represented as p(F0|Dj,F(xiàn)1;Acor,1)=1.A schematic diagram of random failure state transition is shown in Fig. 6.

    From the above, the state transition matrix on Eiis represented by p(Si+1|Si,Ai), where Si+1denotes the aero-engine state on Ei+1, Sidenotes the aero-engine state on Ei, and Aidenotes the performed maintenance action on Ei. Different sub-states would transfer according to different modes, as illustrated above.

    3.4. Total cost and optimization

    In reinforcement learning, an agent chooses maintenance actions according to action costs.44,45In the proposed optimization algorithm, the optimal objective is to obtain a maintenance policy whose total cost is the minimum in the long run.The maintenance cost on decision epoch Ekis calculated by

    where ckdenotes the maintenance cost on decision epoch Ek;Cope,kdenotes the operating cost; Crep,k,Crec,k, Ccor,k, and Cotherdenote the LLP replacement cost, the performance recovery cost,the corrective maintenance cost,and other costs,respectively.

    Fig. 6 Schematic diagram of random failure state transition.

    In general, a good-performance-state aero-engine would have a better economic efficiency and a lower random failure rate. Thus, Cope,kis determined by the performance state.When an LLP replacement action is performed, material and replacement costs are both counted in Crep,k(Arep,k). LLP replacement costs vary from different LLP replacement actions. The performance recovery cost is represented by Crec,k(Arec,k), and when m >n, Crec,i(Arec,m)>Crec,i(Arec,n).The corrective maintenance cost Ccor,kis counted when a corrective maintenance is performed.

    As the life-cycle maintenance policy is optimized by the proposed optimization algorithm, the future maintenance cost should be counted in optimization. In reinforcement learning,a discount factor is adopted to address long-term optimization.Thus,the optimal objective of the proposed optimization algorithm is denoted as

    where C denotes the discounted future cost, and γ(γ ∈[0,1])denotes the discount factor, representing the relative impact of future action costs.

    In reinforcement learning, when a larger γ is adopted,future action costs would leave a greater impact on maintenance action selection.That is,when γ=0,the optimized policy is shortsighted,and the maintenance action is chosen by the current cost; when γ=1, all the future actions are considered in action selection,which would bring a heavy calculation burden. Thus, a balance between future costs and the calculation burden should be determined. Thus, the discount factor γ should be set as γ ∈(0,1), for example, γ=0.9.

    As the Gauss-Seidel value iteration algorithm is an effective reinforcement learning algorithm, which is widely used in policy optimization, it is adopted to seek for the maintenance action collection whose discounted long-term cost is the minimum.

    4. Numerical experiments of maintenance policy optimization

    Two numerical experiments were used to illustrate the proposed aero-engine life-cycle maintenance policy optimization algorithm in detail. According to the reinforcement learning framework,determinations of aero-engine states,maintenance actions, state transition matrices, and total cost matrices are described firstly.As traditional methods are unable to address the long service-life and the hybrid maintenance strategy synchronously,they were not adopted as the benchmark methods.The reinforcement learning algorithm of Gauss-Seidel value iteration was adopted in the experiments.

    4.1. Aero-engine states

    In the reinforcement learning framework,the aero-engine state should be determined firstly. As the multi-dimensional state space was adopted to represent the aero-engine state, performance, LLP, and random failure states were all considered in the first numerical experiment.

    The EGTM was adopted to represent the aero-engine performance state. For convenience, the EGTM time series were divided into several levels. In a sample fleet, there were three performance recovery actions, including minimal repair,medium repair, and overhaul repair. According to the performance recovery assumptions in Section 4.2,there should be at least five performance levels to fully illustrate the three performance recovery actions. In reinforcement learning, as more performance levels would make the optimization algorithm more complicated, five levels were able to present the aeroengine performance state. In the numerical experiment, the performance state was divided into five levels, denoted as{D1,D2,D3,D4,D5}, from good to bad, where D5denoted the worst performance level. Besides, the all-new performance state was denoted as D0.

    Because performance and random failure states were both transferred by probabilities, the random failure state was regarded as a specific‘‘performance state”.Thus,in the numerical experiment, performance and random failure states were represented by one state-space dimension, denoted as{D0,D1,D2,D3,D4,D5,F(xiàn)}.

    Although there were several LLPs in an aero-engine, for convenience,LLPs with the same life limitation were regarded as the same LLP type. In the first numerical experiment, one LLP type was taken into account,and the LLP state was measured by flight-cycles. Referring to the adopted performance state levels,the LLP state was divided into five levels,denoted as {T1,T2,T3,T4,T5}. Besides, T0denoted the all-new state.Thus, the aero-engine state was represented by a twodimensional state space, denoted as

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state F; Tj(j=0,1,2,3,4,5)denoted the LLP state. The performance and random failure states were denoted as the first dimension; the LLP state was denoted as the second dimension.

    Fig. 7 Schematic diagrams of performance recovery action effects on performance states.

    4.2. Maintenance actions

    In reinforcement learning, the aero-engine state was changed by performed maintenance actions, and maintenance actions should be determined according to the aero-engine state.Firstly, according to traditional and practical matters,46two performance recovery assumptions were made in the numerical experiment as follows:

    (1) An aero-engine could not be recovered to the all-new performance state by any performance recovery actions.

    (2) No maintenance action should be performed when the aero-engine was in the worst performance state.

    According to the sample fleet operation, three actions of minimal repair, medium repair, and overhaul repair were included in performance recovery actions. Thus, three performance recovery actions were adopted in the numerical experiment, denoted as Arec={Arec,1,Arec,2,Arec,3}, where Arec-h1was defined as the action to recover the performance state from Dxto Dx-1, when x-1 >0, or to keep in the performance state Dx,when x=1;Arec,2was defined as the action to recover the performance state from Dxto Dx-2, when x-2 >0, or to recover the performance state from Dxto D1,when 1 ≤x ≤2;Arec,3was defined as the action to recover the performance state from Dxto Dx-3, when x-3 >0, or to recover the performance state from Dxto D1, when 1 ≤x ≤3. Besides, Arec,0denoted no performance recovery action was performed.In the reinforcement learning framework, Fig. 7 shows performance recovery action effects on performance states.

    In the numerical experiment, the random failure state was regarded as a specific ‘‘performance state”. Thus, it was assumed that performance recovery actions could drive the random failure aero-engine back to the working state.

    As one LLP type was considered,LLP replacement actions were denoted as Arep={Arep,0,Arep,1}. Fig. 8 shows the LLP replacement action effect on LLP states.

    From the above, maintenance actions in the numerical experiment were represented by

    4.3. State transition

    Fig. 8 Schematic diagram of the LLP replacement action effect on LLP states.

    Based on the aforementioned methods, performance state Diand transition probability matrix P0were obtained from the sample fleet, as shown in

    Transition matrices of Arec,1, Arec,2, and Arec,3were P1, P2,and P3, denoted as

    where nllpdenoted the number of LLP types, and nrecadenoted the number of performance recovery actions.

    4.4. Maintenance policy optimization

    Based on the aforementioned methods, aero-engine states,maintenance actions, and state transition matrices were all determined. In the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy. The flow diagram of the proposed aero-engine maintenance policy optimization algorithm is shown in Fig. 9.

    Table 2 Transition probability matrix of an LLP replacement action.

    Fig. 9 Flow diagram of maintenance policy optimization.

    Because real cost data was unavailable, a hypothetical cost matrix was adopted in the numerical experiment. In engineering,the cost matrix may change according to the actual maintenance cost, and it would not distort the analysis of simulation results. In the numerical experiment, the LLP replacement cost Cllpwas assumed to be 350, while the performance recovery costs Crep,1, Crep,2, and Crep,3were assumed to be 300, 350, and 500. As an aero-engine has long service-life,future maintenance actions should be fully considered in lifecycle maintenance policy optimization.Thus,a larger discount factor γ=0.9 was adopted.In contrast to the Jacobi value iteration algorithm, the Gauss-Seidel value iteration algorithm has a faster convergence ability. Thus, in the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the aero-engine maintenance policy.

    As the aero-engine state was represented by a twodimensional space, the optimal maintenance policy was presented by a two-dimensional policy map, shown in Fig. 10.

    Optimal maintenance actions on each decision epoch were shown in the policy map,and decision epochs were represented by aero-engine states. In the maintenance policy map of Fig. 10, the LLP state was regarded as the ordinate, and the performance state was regarded as the abscissa. Different maintenance actions were presented in different colors and shapes. In the legend, A0denoted action { Arep,0,Arec,0} ; A1denoted action { Arep,0,Arec,1} ; A2denoted action{ Arep,0,Arec,2} ;A4denoted action{ Arep,1,Arec,0} ;A6denoted action { Arep,1,Arec,2} . In engineering, a maintenance policy could be obtained according to the aero-engine state.

    Fig. 10 Maintenance policy map of the numerical experiment.

    4.5. Algorithm performance analysis

    In this section,a more complex numerical experiment was conducted to illustrate the proposed optimization algorithm with more detailed instructions. In engineering, an aero-engine is usually composed of more than one LLP type. Some LLP states were measured by flight-hour, different from the one in the first numerical experiment. Thus, a more complex numerical experiment with two LLP types was conducted.To distinguish the two numerical experiments,the first numerical experiment was named as Experiment 1,and the numerical experiment in this section was named as Experiment 2.

    Table 3 Transition matrices of no LLP replacement action.

    Table 4 Transition probability matrices of LLP1 replacement actions.

    4.5.1. Aero-engine states

    Two LLP types were considered in Experiment 2,and the two LLP states were measured by different units: one was flightcycle and the other was flight-hour.Thus,the aero-engine state was denoted as a three-dimensional state space. The three dimensions included the performance state,the LLP state represented by flight-cycle(LLP1), and the LLP state represented by flight-hour (LLP2).

    Corresponding to Experiment 1, in addition to all-new states T0and L0, the LLP1 state was divided into two levels,denoted as {T0,T1,T2};the LLP2 state was divided into three levels, denoted as {L0,L1,L2,L3}. Same as in Experiment 1,the performance state was represented by five levels, and the random failure state was regarded as a specific ‘‘performance state”. Thus, the aero-engine state space was represented by

    where Di(i=0,1,2,3,4,5) denoted the performance state; D6denoted the random failure state; Tjdenoted the LLP1 state;Lkdenoted the LLP2 state.

    4.5.2. Maintenance actions and state transition

    In Experiment 2, LLP1 and LLP2 replacement actions were denoted as ArepT={ArepT,0,ArepT,1} and ArepL={ArepL,0,ArepL,1}. Performance recovery actions were the same as those in Experiment 1. Performance recovery assumptions were also available.Maintenance actions in Experiment 2 were represented by

    Similar to LLP1, the LLP2 state would transfer as the flight-hour increasing, and transition probabilities were unavailable for LLP2. However, aero-engine state transition matrices were changed by LLP2.

    On Em, action Am={ Arec,i,ArepT,0,ArepL,0|i=0,1,2,3}denoted no LLP replaced, and transition matrices are presented in Table 3, denoted as p(S,Arec,i,ArepT,0,ArepL,0|i=0,1,2,3)= p3{PF,i,,PN,PR,i|i=0,1,2,3}. The concrete forms of PF,, and PRwere the same as those in Experiment 1.

    State transition matrices of Am={ Arec,i,ArepT,1,ArepL,0|i=0,1,2,3} are presented in Table 4, denoted as p(S,Arec,i,ArepT,1,ArepL,0|i = 0,1,2,3) = p4{ PF,i,,PN,PR,i|i = 0,1,2,3} .

    State transition matrices of Am={Arec,i,ArepT,0,ArepL,1|i=0,1,2,3} are presented in Table 5, denoted as p(S,Arec,i,ArepT,0,ArepL,1|i = 0,1,2,3) = p5{PF,i,,PN,PR,i|i = 0,1,2,3}.

    State transition matrices of Am={Arec,i,ArepT,1,ArepL,1|i=0,1,2,3} are presented in Table 6, denoted as p(S,Arec,i,ArepT,1,ArepL,1|i = 0,1,2,3) = p6{PF,i,,PN,PR,i|i = 0,1,2,3}.

    4.5.3. Maintenance policy optimization

    In Experiment 2, the reinforcement learning algorithm of Gauss-Seidel value iteration was also adopted to optimize the maintenance policy. Hypothetical costs were adopted,and the LLP1 replacement cost Cllp,1was assumed to be 300 while the LLP2 replacement cost Cllp,2was assumed to be 600. The three performance recovery action costs were assumed to be 200, 500, and 800, respectively. The discount factor was set as γ=0.9. Because the aero-engine state was represented by a three-dimensional space, including the LLP1, LLP2, and performance states, the optimal maintenance policy was represented by a three-dimensional policy map, shown in Fig. 11.

    Table 6 Transition probability matrices of LLP1 and LLP2 replacement actions.

    Fig. 11 Maintenance policy map of Experiment 2.

    Table 7 Algorithm information of two experiments.

    In the three-dimensional policy map,the LLP1,LLP2,and performance states were regarded as x axis, z axis, and y axis,respectively.The maintenance actions were presented in different colors and shapes.In the legend of the maintenance policy map in Fig. 11, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action{Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action {Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.Based on the policy map, a maintenance policy was obtained according to the aero-engine state.

    4.5.4. Algorithm analysis

    In aforementioned two numerical experiments, a twodimensional state space was adopted in Experiment 1, while a three-dimensional state space was adopted in Experiment 2.It was obvious that state transition matrices of Experiment 2 were more complicated. Thus, the algorithm would become complex as the state number increased. The algorithm information of the two numerical experiments is shown in Table 7.

    In the aforementioned numerical experiments,performance and random failure states were defined as probability states,because they transferred by probabilities. LLP states did not transfer by probabilities, and were defined as definite states.As shown in the two numerical experiments, state transition matrix forms were impacted by definite states. Transition probabilities were impacted by probability states.In reinforcement learning, a larger state space would make the algorithm more complicated, and more iterations were needed to seek for the optimal policy. The impact of aero-engine state space complexity on the algorithm was obvious.

    As the discount factor is an important coefficient in reinforcement learning, the discount factor impact was analyzed by contrast experiments, which were based on Experiment 1.In addition to the discount factor, the other parameters were all the same as those in Experiment 1. Maintenance policy maps of discount factor analysis are shown in subgraphs of Fig. 12.

    Fig. 12 Policy maps of discount factor analysis in Example 1.

    In the policy maps,different maintenance actions were represented by different colors and shapes. In the legend of Fig. 12, A0denoted action {Arep,0,Arec,0}; A1denoted action{Arep,0,Arec,1}; A2denoted action {Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A6denoted action {Arep,1,Arec,2}. The analysis experiments showed that when the discount factor was set as γ >0.43,the optimal policy maps were the same as those in Fig. 10;when the discount factor was set as γ <0.35, the optimal policy maps were the same as those in Fig. 12(c). In Fig. 12, discount factors were set as γ = 0.43, γ = 0.41, and γ = 0.35,respectively. The analysis experiments showed that when a smaller discount factor was adopted, more low-cost maintenance actions were adopted in the optimal maintenance policy.It was consistent with the aforementioned analysis of the discount factor.

    As hypothetical maintenance costs were adopted in numerical experiments, cost impacts were analyzed. Based on the aforementioned experiments, the medium performance recovery cost was set as Crep,2=350, and it was regarded as the benchmark cost. Maintenance policy maps of different cost ratios are shown in subgraphs of Figs. 13 and 14.

    Fig. 13 Policy maps of performance recovery cost analysis in Example 1.

    In the legend of Fig. 13, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A3denoted action {Arep,0,Arec,3}; A4denoted action {Arep,1,Arec,0}; A5denoted action {Arep,1,Arec,1}; A7denoted action {Arep,1,Arec,3}. In the legend of Fig. 14, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A3denoted action {Arec,3,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A8denoted action {Arec,3,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1};A14denoted action{Arec,3,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}.

    Fig. 14 Policy maps of performance recovery cost analysis in Example 2.

    Fig. 15 Policy maps of LLP replacement cost analysis in Example 1.

    As shown in Figs.13(a)and 14(a),showed the optimal policy when Crec,1decreased; Figs. 13(b) and 14(b) showed the optimal policy when Crec,3decreased; Figs. 13(c) and 14(c)showed the optimal policy when Crec,1and Crec,3decreased simultaneously. As shown in Figs. 13(c) and 14(c), when Crec,1and Crec,3decreased simultaneously, optimal policy changes were not obvious.

    The impact of the LLP replacement cost on the optimal maintenance policy was analyzed by contrast experiments.Experiment results showed that the optimal maintenance policy would not change as the LLP replacement cost increasing.However,LLP replacement action times would increase as the LLP replacement cost decreasing.Optimal policy maps of LLP replacement cost analysis are shown in Figs. 15-17.

    In the legend of Fig. 15, A0denoted action {Arep,0,Arec,0};A1denoted action {Arep,0,Arec,1}; A2denoted action{Arep,0,Arec,2};A4denoted action {Arep,1,Arec,0}; A6denoted action {Arep,1,Arec,2}. In the legends of Figs. 16 and 17, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action{Arec,1,ArepT,0,ArepL,0};A2denoted action {Arec,2,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action{Arec,1,ArepT,1,ArepL,0}; A7denoted action {Arec,2,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A13denoted action {Arec,2,ArepT,1,ArepL,1}; A15denoted action{Arec,0,ArepT,1,ArepL,1}.

    As shown in Fig. 15, the LLP replacement cost decreased from subgraph (a) to (c). In Figs. 16 and 17, LLP1 and LLP2 replacement costs decreased from subgraph (a) to (b).It was shown that LLP replacement times would increase as the LLP replacement cost decreasing, and policy changes appeared on interface decision epochs.

    In the aforementioned experiments, the impact of the LLP residual life was not considered. Thus, based on the assumption that the random failure probability would increase as the LLP residual life decreasing, contrast experiments were performed to analyze the optimization algorithm.Optimal policy maps are shown in Fig. 18.

    Fig. 16 Policy maps of LLP1 replacement cost analysis in Example 2.

    Fig. 17 Policy maps of LLP2 replacement cost analysis in Example 2.

    In the legend of Fig. 18, A0denoted action {Arec,0,ArepT,0,ArepL,0}; A1denoted action {Arec,1,ArepT,0,ArepL,0}; A4denoted action {Arec,0,ArepT,1,ArepL,0}; A5denoted action {Arec,0,ArepT,0,ArepL,1}; A6denoted action {Arec,1,ArepT,1,ArepL,0}; A9denoted action {Arec,1,ArepT,0,ArepL,1}; A12denoted action{Arec,1,ArepT,1,ArepL,1}; A15denoted action {Arec,0,ArepT,1,ArepL,1}. Based on Experiment 2, according to the residual life of the elder LLP, random failure probabilities were enhanced by 5%, 10%, and 15% respectively from subgraph (a) to (c)of Fig. 18. Policy maps showed no variation. Thus, the LLP residual life may not affect the optimal maintenance policy.

    5. Conclusions

    Based on the reinforcement learning approach,an aero-engine life-cycle maintenance policy optimization algorithm was proposed, which was able to address the long service-life and the hybrid maintenance strategy synchronously. To address the hybrid maintenance strategy,the multi-dimensional state space was adopted to represent the aero-engine state. Based on the reinforcement learning framework, the Gauss-Seidel value iteration algorithm was adopted to optimize the life-cycle maintenance policy.

    Compared with traditional optimization methods, the optimal maintenance policy was used to indicate when and how to repair an aero-engine, taking the place of maintenance intervals and work-scopes in traditional methods.Meanwhile, the aero-engine long service-life, the hybrid maintenance strategy, and random factor destabilization were all addressed by the proposed optimization algorithm.Because few historical data was available for training the pre-specified optimization model of the aero-engine lifecycle maintenance policy, the reinforcement learning approach provided an appropriate way. In the reinforcement learning framework, the aero-engine state space, maintenance actions, and state transition matrices were determined according to aero-engine real-life operation. The Gauss-Seidel value iteration algorithm was employed to solve the long-term decision-making problem. The proposed optimization algorithm would help in making a wiser aero-engine life-cycle maintenance policy, resulting in a lower life-cycle maintenance cost. Two numerical experiments and algorithm analyses were employed to illustrate the proposed optimization algorithm in detail.

    As real aero-engine maintenance cost data was unavailable,hypothetical data was adopted in the numerical experiments.In future studies, maintenance cost calculation methods deserve further attention to improve the applicability of the proposed optimization algorithm.

    Fig. 18 Policy maps with LLP lifetime impact on transition probability.

    Acknowledgments

    The authors thank anonymous reviewers for their critical and constructive review of the manuscript. This work was cosupported by the Key National Natural Science Foundation of China (No. U1533202), the Civil Aviation Administration of China (No. MHRD20150104), and the Shandong Independent Innovation and Achievements Transformation Fund,China (No. 2014CGZH1101).

    中文字幕色久视频| 亚洲欧美一区二区三区黑人| 精品高清国产在线一区| 黄色视频,在线免费观看| 真人做人爱边吃奶动态| 国产精品av久久久久免费| 色综合婷婷激情| 人妻一区二区av| 50天的宝宝边吃奶边哭怎么回事| 国产欧美日韩一区二区三| 手机成人av网站| 我的亚洲天堂| 久久久国产一区二区| 精品免费久久久久久久清纯 | 亚洲国产精品sss在线观看 | 亚洲一卡2卡3卡4卡5卡精品中文| 99在线人妻在线中文字幕 | 亚洲人成电影观看| 亚洲精品美女久久久久99蜜臀| 亚洲 国产 在线| 黄色怎么调成土黄色| 自拍欧美九色日韩亚洲蝌蚪91| 中文欧美无线码| 亚洲精品中文字幕一二三四区| 亚洲久久久国产精品| 天天躁狠狠躁夜夜躁狠狠躁| 女人爽到高潮嗷嗷叫在线视频| 高清在线国产一区| 国产成人一区二区三区免费视频网站| 国产成人精品久久二区二区91| 另类亚洲欧美激情| 日韩中文字幕欧美一区二区| 久久久久久久久免费视频了| 精品久久蜜臀av无| 最新在线观看一区二区三区| 国精品久久久久久国模美| 国产精品 欧美亚洲| 精品国产一区二区久久| 国产欧美日韩一区二区精品| bbb黄色大片| 久久精品亚洲av国产电影网| 国产一卡二卡三卡精品| 亚洲熟妇熟女久久| 最近最新中文字幕大全免费视频| 中亚洲国语对白在线视频| 欧美在线一区亚洲| 国产成人av教育| 多毛熟女@视频| 亚洲情色 制服丝袜| 色综合婷婷激情| 看片在线看免费视频| 美女福利国产在线| 大片电影免费在线观看免费| 嫁个100分男人电影在线观看| 99re在线观看精品视频| 亚洲专区国产一区二区| 国产乱人伦免费视频| 久久精品亚洲熟妇少妇任你| 在线观看66精品国产| 国产无遮挡羞羞视频在线观看| 亚洲伊人色综图| 乱人伦中国视频| 999精品在线视频| 国产精品一区二区精品视频观看| 手机成人av网站| 老熟妇仑乱视频hdxx| 国产一区有黄有色的免费视频| 国产不卡av网站在线观看| 日韩欧美一区二区三区在线观看 | 中亚洲国语对白在线视频| 黄片小视频在线播放| av视频免费观看在线观看| 人妻 亚洲 视频| 成年人免费黄色播放视频| 亚洲精品美女久久久久99蜜臀| 日本五十路高清| 国产野战对白在线观看| 国产精品国产av在线观看| 亚洲精品一二三| 国产一区二区三区在线臀色熟女 | 亚洲国产精品一区二区三区在线| 久久久久国内视频| 99热国产这里只有精品6| 日本wwww免费看| 欧美日韩视频精品一区| 日韩三级视频一区二区三区| 美女高潮喷水抽搐中文字幕| 一二三四社区在线视频社区8| 国产精品久久久久久精品古装| 亚洲第一欧美日韩一区二区三区| 久久精品91无色码中文字幕| 午夜免费观看网址| 又黄又粗又硬又大视频| 麻豆av在线久日| av免费在线观看网站| 制服人妻中文乱码| 伦理电影免费视频| 国产片内射在线| 欧洲精品卡2卡3卡4卡5卡区| 老司机影院毛片| 精品乱码久久久久久99久播| 变态另类成人亚洲欧美熟女 | 香蕉久久夜色| 无人区码免费观看不卡| av超薄肉色丝袜交足视频| 国产亚洲精品久久久久5区| 免费高清在线观看日韩| 午夜福利免费观看在线| 国产一区二区三区综合在线观看| 一级黄色大片毛片| 精品第一国产精品| av网站在线播放免费| 99在线人妻在线中文字幕 | 妹子高潮喷水视频| 成人永久免费在线观看视频| 欧美中文综合在线视频| 欧美成人免费av一区二区三区 | 一区在线观看完整版| 成人免费观看视频高清| 免费黄频网站在线观看国产| 欧美日韩国产mv在线观看视频| 久99久视频精品免费| 国产一区二区三区在线臀色熟女 | 国产精品国产高清国产av | 亚洲国产看品久久| 黄色 视频免费看| 免费在线观看影片大全网站| 国产欧美日韩一区二区三区在线| 在线观看66精品国产| 日日夜夜操网爽| 中出人妻视频一区二区| 国产av又大| 亚洲人成77777在线视频| 亚洲专区中文字幕在线| 好看av亚洲va欧美ⅴa在| 精品少妇久久久久久888优播| 高清av免费在线| 免费久久久久久久精品成人欧美视频| 黑人猛操日本美女一级片| 亚洲人成电影免费在线| 叶爱在线成人免费视频播放| 黄片小视频在线播放| 亚洲国产毛片av蜜桃av| 亚洲三区欧美一区| 成人永久免费在线观看视频| av一本久久久久| 一进一出抽搐动态| 女人精品久久久久毛片| 成年女人毛片免费观看观看9 | 亚洲一区二区三区欧美精品| 国产一区二区三区综合在线观看| 黄色 视频免费看| 中文字幕最新亚洲高清| 午夜福利在线观看吧| 国产欧美亚洲国产| 国产一区二区三区综合在线观看| 国产精品免费一区二区三区在线 | 国产又色又爽无遮挡免费看| 国产主播在线观看一区二区| 精品人妻1区二区| 欧美 亚洲 国产 日韩一| 脱女人内裤的视频| 精品一区二区三区四区五区乱码| 一级片免费观看大全| 亚洲精品一二三| 久久狼人影院| 久久久久久久国产电影| 午夜福利视频在线观看免费| 国产不卡一卡二| 欧美成人午夜精品| 国产精品乱码一区二三区的特点 | 久久精品国产a三级三级三级| 午夜激情av网站| 中出人妻视频一区二区| 成人手机av| 国产精华一区二区三区| 国产精品偷伦视频观看了| 亚洲色图 男人天堂 中文字幕| 精品一区二区三区四区五区乱码| 大码成人一级视频| 老司机午夜福利在线观看视频| 中文字幕人妻熟女乱码| 狠狠狠狠99中文字幕| 欧美黑人欧美精品刺激| 久久天堂一区二区三区四区| 国产一区在线观看成人免费| av线在线观看网站| 一二三四在线观看免费中文在| 亚洲欧美日韩另类电影网站| 老司机靠b影院| 乱人伦中国视频| 精品久久久久久,| 女人被躁到高潮嗷嗷叫费观| 母亲3免费完整高清在线观看| 精品一区二区三卡| 午夜日韩欧美国产| 成人精品一区二区免费| xxx96com| 国产精品 欧美亚洲| 自拍欧美九色日韩亚洲蝌蚪91| 免费不卡黄色视频| 成人18禁在线播放| 最新美女视频免费是黄的| 国产精品久久久久成人av| 日本黄色视频三级网站网址 | 视频区欧美日本亚洲| 午夜福利免费观看在线| 人人妻人人澡人人爽人人夜夜| 1024香蕉在线观看| 黄片小视频在线播放| 午夜福利乱码中文字幕| avwww免费| videos熟女内射| 91成人精品电影| 天天躁夜夜躁狠狠躁躁| 亚洲成人免费av在线播放| 一区二区日韩欧美中文字幕| 久久久精品免费免费高清| 99精品欧美一区二区三区四区| 国内毛片毛片毛片毛片毛片| 女性生殖器流出的白浆| 久久精品国产清高在天天线| 天天影视国产精品| 亚洲少妇的诱惑av| 精品亚洲成国产av| 飞空精品影院首页| 亚洲精品在线美女| 一进一出抽搐gif免费好疼 | 黄色毛片三级朝国网站| 国产精品国产高清国产av | 日韩熟女老妇一区二区性免费视频| 免费在线观看完整版高清| 大香蕉久久网| 成年动漫av网址| 99re在线观看精品视频| 纯流量卡能插随身wifi吗| 美女午夜性视频免费| 黄片小视频在线播放| 国内毛片毛片毛片毛片毛片| 制服诱惑二区| 亚洲av成人一区二区三| 人人妻,人人澡人人爽秒播| 天天添夜夜摸| 国产成人一区二区三区免费视频网站| 看片在线看免费视频| 少妇粗大呻吟视频| 视频区欧美日本亚洲| 亚洲五月天丁香| 国产精品一区二区免费欧美| 黄色 视频免费看| 在线天堂中文资源库| x7x7x7水蜜桃| 亚洲第一欧美日韩一区二区三区| av国产精品久久久久影院| 999久久久精品免费观看国产| 国产蜜桃级精品一区二区三区 | 亚洲色图 男人天堂 中文字幕| 欧美日韩亚洲国产一区二区在线观看 | 精品一品国产午夜福利视频| 一进一出好大好爽视频| 国产亚洲精品久久久久5区| 久久精品国产亚洲av高清一级| 色94色欧美一区二区| 精品久久久久久久毛片微露脸| 露出奶头的视频| 最近最新免费中文字幕在线| 日韩大码丰满熟妇| 性色av乱码一区二区三区2| 天堂中文最新版在线下载| 国产色视频综合| 国产成人啪精品午夜网站| 欧美日韩精品网址| 国产精品香港三级国产av潘金莲| 视频区欧美日本亚洲| 午夜精品在线福利| 丰满人妻熟妇乱又伦精品不卡| 制服人妻中文乱码| 最新美女视频免费是黄的| 日日摸夜夜添夜夜添小说| 老司机深夜福利视频在线观看| 国产精品一区二区精品视频观看| 女同久久另类99精品国产91| 国产精品综合久久久久久久免费 | 十八禁高潮呻吟视频| 国产不卡av网站在线观看| 免费人成视频x8x8入口观看| 精品久久久久久电影网| 久久久久久久久免费视频了| 亚洲五月色婷婷综合| 国产男靠女视频免费网站| 国产精品久久久av美女十八| 欧美在线黄色| 欧美午夜高清在线| 99热只有精品国产| 91成年电影在线观看| 久久人妻福利社区极品人妻图片| 午夜福利免费观看在线| 国产精品偷伦视频观看了| 精品久久久久久电影网| 国产在线精品亚洲第一网站| 久热爱精品视频在线9| 国产av一区二区精品久久| 夜夜爽天天搞| 欧美日韩福利视频一区二区| 三级毛片av免费| 精品午夜福利视频在线观看一区| 久久ye,这里只有精品| 亚洲情色 制服丝袜| 免费一级毛片在线播放高清视频 | 国产av一区二区精品久久| 国产一区二区三区在线臀色熟女 | 久久天躁狠狠躁夜夜2o2o| 狂野欧美激情性xxxx| 99精品在免费线老司机午夜| 天天躁日日躁夜夜躁夜夜| 777久久人妻少妇嫩草av网站| 亚洲精品中文字幕在线视频| 亚洲第一av免费看| 国产精品二区激情视频| 亚洲av日韩在线播放| 精品熟女少妇八av免费久了| 亚洲avbb在线观看| 免费一级毛片在线播放高清视频 | 亚洲九九香蕉| 精品一区二区三区四区五区乱码| 亚洲精品久久午夜乱码| 少妇粗大呻吟视频| 亚洲熟妇中文字幕五十中出 | www.999成人在线观看| 久久精品国产亚洲av高清一级| 欧美日韩一级在线毛片| 中文字幕最新亚洲高清| 女人爽到高潮嗷嗷叫在线视频| 麻豆乱淫一区二区| 日韩精品免费视频一区二区三区| 国产免费现黄频在线看| 一级片免费观看大全| 国产色视频综合| 亚洲精品国产一区二区精华液| 色综合婷婷激情| 91字幕亚洲| 亚洲人成77777在线视频| 在线观看免费午夜福利视频| 色综合欧美亚洲国产小说| 国产精品久久电影中文字幕 | 国产精品乱码一区二三区的特点 | av视频免费观看在线观看| 天堂√8在线中文| 久久九九热精品免费| 精品一区二区三区av网在线观看| 亚洲欧美激情在线| 欧美黑人欧美精品刺激| 欧美日韩福利视频一区二区| 亚洲 欧美一区二区三区| 九色亚洲精品在线播放| 国产深夜福利视频在线观看| 亚洲,欧美精品.| 叶爱在线成人免费视频播放| 超碰成人久久| 身体一侧抽搐| 18禁国产床啪视频网站| 最近最新中文字幕大全免费视频| 国产激情久久老熟女| 国产亚洲精品久久久久久毛片 | 性少妇av在线| 在线十欧美十亚洲十日本专区| 久久亚洲精品不卡| 免费看十八禁软件| 怎么达到女性高潮| 嫩草影视91久久| 欧美日韩中文字幕国产精品一区二区三区 | 亚洲男人天堂网一区| а√天堂www在线а√下载 | 久久婷婷成人综合色麻豆| 亚洲精品一二三| 亚洲精品自拍成人| 国产97色在线日韩免费| 美女福利国产在线| 国产99白浆流出| 亚洲熟妇熟女久久| 免费在线观看黄色视频的| 在线观看舔阴道视频| 在线视频色国产色| 啪啪无遮挡十八禁网站| 国产精品一区二区在线不卡| 国产精品1区2区在线观看. | 国产一卡二卡三卡精品| 18禁美女被吸乳视频| 欧美成人免费av一区二区三区 | 婷婷成人精品国产| 国产av一区二区精品久久| 国产精品av久久久久免费| 国产欧美日韩一区二区精品| 高潮久久久久久久久久久不卡| 在线观看www视频免费| 国产午夜精品久久久久久| 免费在线观看亚洲国产| 又黄又爽又免费观看的视频| 久久国产乱子伦精品免费另类| 黄色视频,在线免费观看| 午夜老司机福利片| 男女高潮啪啪啪动态图| 国产精品一区二区免费欧美| 热99国产精品久久久久久7| 欧美激情久久久久久爽电影 | 国产精品久久视频播放| 国产精品 欧美亚洲| 超碰97精品在线观看| 国产一区二区激情短视频| 成人国语在线视频| 欧美精品啪啪一区二区三区| 国产一区有黄有色的免费视频| 叶爱在线成人免费视频播放| 国产极品粉嫩免费观看在线| 国产成人免费无遮挡视频| 免费在线观看影片大全网站| 亚洲av美国av| 亚洲国产欧美一区二区综合| 91麻豆av在线| av不卡在线播放| 午夜精品国产一区二区电影| 黄色视频不卡| 99国产精品免费福利视频| 中文字幕人妻丝袜制服| av福利片在线| 精品国内亚洲2022精品成人 | 国产亚洲av高清不卡| 久久精品aⅴ一区二区三区四区| 男女之事视频高清在线观看| e午夜精品久久久久久久| 侵犯人妻中文字幕一二三四区| 亚洲精品一二三| 国产精品秋霞免费鲁丝片| 又黄又爽又免费观看的视频| 深夜精品福利| 久久这里只有精品19| 女人爽到高潮嗷嗷叫在线视频| av天堂在线播放| 久久国产精品大桥未久av| 色婷婷av一区二区三区视频| 午夜91福利影院| 国产成人系列免费观看| 黄频高清免费视频| 中出人妻视频一区二区| 中文字幕精品免费在线观看视频| 亚洲伊人色综图| 久久中文字幕人妻熟女| 夫妻午夜视频| 久久精品国产a三级三级三级| 电影成人av| 久久国产亚洲av麻豆专区| 日本vs欧美在线观看视频| 精品久久久久久电影网| 黄色毛片三级朝国网站| 自线自在国产av| 视频区欧美日本亚洲| 日本vs欧美在线观看视频| 国产成+人综合+亚洲专区| 99国产极品粉嫩在线观看| 欧美激情高清一区二区三区| 下体分泌物呈黄色| 亚洲第一青青草原| www.精华液| 波多野结衣av一区二区av| 免费不卡黄色视频| 欧美+亚洲+日韩+国产| 国产av又大| 久久香蕉精品热| 欧美精品人与动牲交sv欧美| 久久国产亚洲av麻豆专区| 亚洲男人天堂网一区| 亚洲avbb在线观看| 18禁观看日本| 夜夜爽天天搞| 亚洲午夜理论影院| 亚洲精品美女久久av网站| 欧美国产精品一级二级三级| 黑人巨大精品欧美一区二区蜜桃| 18禁美女被吸乳视频| 男女高潮啪啪啪动态图| 黄色 视频免费看| 老汉色av国产亚洲站长工具| 操美女的视频在线观看| 又紧又爽又黄一区二区| 亚洲av日韩精品久久久久久密| 久久久国产成人精品二区 | 国产精品一区二区免费欧美| 亚洲 国产 在线| 免费看十八禁软件| 亚洲午夜理论影院| 亚洲,欧美精品.| 自拍欧美九色日韩亚洲蝌蚪91| 日韩免费av在线播放| 这个男人来自地球电影免费观看| 精品国产乱子伦一区二区三区| 男女午夜视频在线观看| 69av精品久久久久久| 久久久久久久久免费视频了| 天天躁夜夜躁狠狠躁躁| 亚洲av第一区精品v没综合| 国产高清国产精品国产三级| 91字幕亚洲| 免费人成视频x8x8入口观看| 中文字幕av电影在线播放| videos熟女内射| 国产成人影院久久av| 国产激情欧美一区二区| 99re在线观看精品视频| 成年人黄色毛片网站| 搡老熟女国产l中国老女人| a级片在线免费高清观看视频| 国产激情久久老熟女| 国产精品香港三级国产av潘金莲| tocl精华| 国产亚洲精品久久久久久毛片 | 亚洲欧美日韩高清在线视频| 亚洲片人在线观看| 中出人妻视频一区二区| 天堂俺去俺来也www色官网| 欧美日韩亚洲综合一区二区三区_| 熟女少妇亚洲综合色aaa.| 精品国产亚洲在线| 久久中文字幕一级| 精品第一国产精品| 亚洲国产精品sss在线观看 | 新久久久久国产一级毛片| 一级片免费观看大全| 亚洲国产欧美网| 另类亚洲欧美激情| 视频区图区小说| 中文亚洲av片在线观看爽 | 9色porny在线观看| 国产xxxxx性猛交| a在线观看视频网站| 一二三四在线观看免费中文在| 亚洲国产精品一区二区三区在线| 亚洲午夜理论影院| 国产精品九九99| xxxhd国产人妻xxx| 亚洲第一青青草原| 多毛熟女@视频| 女人高潮潮喷娇喘18禁视频| 国产黄色免费在线视频| 99久久99久久久精品蜜桃| 宅男免费午夜| 大片电影免费在线观看免费| 亚洲 国产 在线| 高清毛片免费观看视频网站 | 久久午夜亚洲精品久久| 老熟妇乱子伦视频在线观看| 日韩熟女老妇一区二区性免费视频| 丰满迷人的少妇在线观看| 精品一区二区三卡| 18禁裸乳无遮挡动漫免费视频| 午夜视频精品福利| 丝袜人妻中文字幕| 欧美人与性动交α欧美软件| videosex国产| 一级毛片精品| 亚洲精品粉嫩美女一区| 精品少妇久久久久久888优播| 久久中文看片网| www.熟女人妻精品国产| 国产蜜桃级精品一区二区三区 | 飞空精品影院首页| 一边摸一边抽搐一进一小说 | 欧美黑人欧美精品刺激| 正在播放国产对白刺激| 一区福利在线观看| 色婷婷av一区二区三区视频| 精品电影一区二区在线| 久久香蕉激情| 妹子高潮喷水视频| 最新的欧美精品一区二区| a级毛片黄视频| 一夜夜www| 欧美国产精品va在线观看不卡| 国产亚洲精品久久久久5区| 国产亚洲精品一区二区www | www.熟女人妻精品国产| 在线观看免费视频网站a站| 国产精品免费大片| 亚洲国产看品久久| 亚洲成a人片在线一区二区| ponron亚洲| 制服诱惑二区| 成人特级黄色片久久久久久久| 欧美在线一区亚洲| 在线十欧美十亚洲十日本专区| 叶爱在线成人免费视频播放| 精品国产国语对白av| 久久久国产精品麻豆| 国产精品电影一区二区三区 | 欧洲精品卡2卡3卡4卡5卡区| 美女午夜性视频免费| 欧美一级毛片孕妇| 日本欧美视频一区| 亚洲第一欧美日韩一区二区三区| 亚洲欧美色中文字幕在线| 校园春色视频在线观看| 久热这里只有精品99| 久久久久精品国产欧美久久久| 午夜免费观看网址| 国产成人av激情在线播放| 日韩三级视频一区二区三区| 美女福利国产在线| 日韩视频一区二区在线观看| 色播在线永久视频| 国产精品自产拍在线观看55亚洲 | 亚洲国产看品久久| 欧美黄色淫秽网站| 国产真人三级小视频在线观看| 中出人妻视频一区二区| 久久久国产欧美日韩av| 夜夜夜夜夜久久久久|