Remaining useful life prediction for engineering systems under dynamic operational conditions:A semi-Markov decision process-based approach

Diyin TANG,Jinrong CAO,Jinsong YU,b,*

aSchool of Automation Science and Electrical Engineering,Beihang University,Beijing 100083,China

bCollaborative Innovation Center of Advanced Aero-Engine,Beijing 100083,China

KEYWORDS Condition-specific failure threshold;Degradation modeling;Dynamic operational conditions;Remaining useful life;Semi-Markov decision process

Abstract For critical engineering systems such as aircraft and aerospace vehicles,accurate Remaining Useful Life(RUL)prediction not only means cost saving,but more importantly,is of great significance in ensuring system reliability and preventing disaster.RUL is affected not only by a system's intrinsic deterioration,but also by the operational conditions under which the system is operating.This paper proposes an RUL prediction approach to estimate the mean RUL of a continuously degrading system under dynamic operational conditions and subjected to condition monitoring at short equi-distant intervals.The dynamic nature of the operational conditions is described by a discrete-time Markov chain,and their influences on the degradation signal are quanti fi ed by degradation rates and signal jumps in the degradation model.The uniqueness of our proposed approach is formulating the RUL prediction problem in a semi-Markov decision process framework,by which the system mean RUL can be obtained through the solution to a limited number of equations.To extend the use of our proposed approach in real applications,different failure standards according to different operational conditions are also considered.The application and effectiveness of this approach are illustrated by a turbofan engine dataset and a comparison with existing results for the same dataset.

1.Introduction

Critical engineering systems,such as engines and turbines in aircraft,are required to run safely for their whole lifetime.However,no matter how reliable they are,deterioration of quality and performance due to aging and operating environment will gradually impair them,which will lead to their ultimate failures.Predicting the Remaining Useful Life(RUL)of engineering systems has been an effective strategy to prevent disastrous failures,and more than that,this prediction is also important to aid health management actions such as maintenance and logistical support.

In RUL prediction problems,the degradation model is of vital importance.In existing literature,many degradation models assume that operational conditions are temporallyinvariant,or have no effect on the deterioration and failure process.1The operational conditions we consider include natural environment conditions surrounding a system(e.g.,temperature,relative humidity,pressure)and working conditions under which the system is operating(e.g.,loading,usage,rotatory speed).However,excluding the influences of operational conditions out of degradation modeling is not always appropriate.For example,the loading condition is an important cause for a change of the vibration signal of mechanical equipment.Mechanical equipment with a heavier load usually generates a higher magnitude of vibration signal than that with a lighter load.Therefore,it is usually more appropriate to take operational conditions into account in degradation modeling.

As operational conditions usually change over time during system deterioration,the degradation model should better be prepared to adapt to this change.Some distinguished degradation modeling approaches may have the potential of dealing with changeable operational conditions.They utilize a parameter updating procedure to update the parameters in the degradation model,in order for the model to accommodate the unknown future.For instances,Gebraeel et al.2developed a Bayesian method to update the stochastic parameters of exponential degradation models using real-time condition monitoring information,and Chen et al.3utilized both current observations and future prediction results obtained by a support vector machine to update all system model parameters.More examples can be found in Refs.4-7.These methods provide feasible ways for degradation models to respond to challengesofunknown and changeable future operational conditions.However,since they take no effort on analyzing how a specific operational condition influences deterioration and how operational conditions evolve,they model the random behavior of degradation under changeable operational conditions quite blindly.

In fact,there are many examples in real engineering practices that deterioration can be determined by operational conditions;if we carefully examine the evolutions of these operational conditions,we may find some ways to describe the degradation process with the help of those evolutionary laws of operational conditions.For instances,if the evolutions of future operational conditions follow a deterministic profile,Liao and Tian8developed an enhanced Bayesian updating technique for a Wiener process-based degradation model to accommodate piecewise constant operating conditions.On the other hand,when future operational conditions evolve in a random way,a dynamic law such as the Poisson process and the Markov process can be used to describe them in degradation modeling.Bocchetti et al.9assumed that the wear of cylinder liner is accumulated at each random shock which arrives according to a Poisson shock arrival process.Si et al.10proposed a numerical solution algorithm to estimate the RUL based on a Wiener process-based degradation model which has operation state switches between working and storage states.The switch ofoperation states is characterized by a Continuous-Time Markov Chain(CTMC).The above two examples also respectively represent two types of generally used environmental effects of operational conditions exerted on degradation processes.One is a random shock that creates a certain amount of damage each time it occurs,and the other one is a random change in degradation rates.Bian et al.1combined these two environmental effects in their Wiener processbased degradation model,in which failure rates are determined by specific operational conditions,and upward or downward signal jumpsare induced by transitions of operational conditions.

In this paper,we will focus on the RUL prediction problem for engineering systems subject to dynamic operational conditions.We consider that the condition-monitoring data of a system is collected from various sensors at equi-distant sampling epochs.The degradation information of the system will be excavated from each sensor,and those believed to have a capability of predicting the RUL are selected.We will develop a degradation model to describe the degradation signal of each selected sensor,and then combine their RUL prediction results.In this way,each degradation signal has physical meanings,and the RUL prediction approach is able to select sensors that are truly useful in describing the degradation process of the system.

Similar to that of Bian et al.1,our degradation model is also composed of degradation rates and signal jumps,respectively determined by different operational conditions and operational transitions.Unlike the CTMC assumption for describing the evolutions of operational conditions in Ref.1,we consider an equidistant condition monitoring situation,and the transition law for operational conditions during a sampling interval is assumed to follow a Discrete-Time Markov Chain(DTMC).This modeling approach is appropriate for situations where the sampling interval is short and operational conditions change frequently.Secondly,we consider that failure thresholds are different under different operational conditions.This condition-specific failure mechanism is more realistic and appropriate in many real applications.For example,a deteriorated system may not be able to function under harsh operational conditions,but it may work safely under mild conditions.

To compute the RUL based on the above model and failure mechanism,a suitable computationalmethod is necessary.We will develop a novel approach based on the Semi-Markov Decision Process(SMDP)framework to address this problem.Using this approach,the mean RUL can be obtained by solving a limited number of equations.A wide variety of advanced computational techniques such as stochastic or statistical methods for specific mathematical models11-15,support vector machines3,state filtering methods16,artificial neural networks17,and dynamic Bayesian networks18have been used for estimating the RUL of engineering systems.To our knowledge,there is no reported approach dealing with the condition-specific failure mechanism and using the SMDP framework to estimate the system mean RUL.The practical value of our proposed RUL prediction approach will be illustrated and validated by a case study of a turbofan engine.

The remainder of the paper is organized as follows.In Section 2,a model to describe the evolution of the degradation signal under dynamic operational conditions is developed.Section 3 proposes an RUL prediction approach using a SMDP framework.Section 4 presents the whole RUL prediction procedure using a turbofan engine dataset which is also the PHM08 Challenge Data,and discusses prediction results.Conclusions and future works are given in Section 5.

2.Degradation modeling

In this section,we will present a degradation model to describe the time series process of the degradation signal for any selected sensor subject to condition monitoring at equidistant short operating cycles.The degradation signal is fully determined by dynamic operational conditions which vary stochastically according to a discrete-time Markov chain.

2.1.Model for dynamic operational conditions

Let C（n）be the operational condition observed at sampling epoch tn=nh,where h is the cycle length and n is the cycle number.C（n）can be one and only one state of a set E= ｛1 ，2，...，M｝,where M＜∝is the total number of operational conditions.An operational transition from one operational condition to another happens within the operating cycle,and we assume that the cycle length h is short enough to ensure that only one transition could happen within one cycle.The operational transition is supposed to follow a DTMC with transition probabilities Pkk′given by

The estimation of the one-step transition probabilities of Pkk′is given by

where Nkk′is the total number of transitions from operational condition k to k′in one cycle for the whole training dataset,and Nkis the total number of times that the system is observed under operational condition k.

If the estimated DTMC is an irreducible ergodic Markov chain(see Ref.19),then its limiting probabilitiesexist,and can be calculated by the following system of linear equations:

For a particular situation that limiting probabilities exist,a simple approach can be adopted to evaluate whether a DTMC is suitable to describe the dynamic nature of the operational conditions.The limiting probability πk′of a DTMC is supposed to equal to the long-run proportion of time that the process will be in state k′(see Ref.19).Thus,if the long-run proportion of time that the process is in state k′can be demonstrated to be equal to the limiting probability πk′,a DTMC is very likely to be suitable for this dataset.This proportion of time can be approximated by the frequency Fk′that the process is found in state k′among all available historical data,which is given by

Furthermore,describing the dynamics of operational conditions by a DTMC is not enough for developing our continuous-state degradation model.The exact time when an operational transition happens within one operating cycle is also necessary for degradation modeling.Since in this paper we focus on situations where the cycle length h is short,we assume that an operational transition always occurs at the right end of each cycle.

2.2.Model for degradation signal under dynamic operational conditions

For any selected sensor s,let Y（n）denote the degradation signal observed at sampling epoch tn=nh(n=1，2，...).If operational condition k at time tn=nh is known,the signal is denoted by Yk（n）.

We assume the degradation signal is subjected to two processes.One is gradual degradation under invariant operational conditions,and the other is signal jumps when operational transitions occur.We de fine a rate function μ（k）to denote the rate of degradation when the system is operating under operational condition k.We also de fine a function for signal jumps,denoted by J（k，k′）,which describes the signal's upward or downward jumps at operational transition epochs.For our model,the jump magnitude is determined by the operational condition just before and just after the operational transition.That is,J（k，k′）represents the jump magnitude when an operational transition from operational condition k to k′occurs.We take into account the signal jump because it is typical in many real applications.For example,as Bian et al.1pointed out,in vibration signals,it is very common to observe a downward jump in a signal when the load on the unit is reduced,and an upward jump when the load is increased.Finally,the degradation signal at time tn=nh is given by

where Ajis the jth transition time of the operational conditions,and N（t）is the cumulative number of operational transitions up to tn=nh.Since we neglect the time duration from the occurrence to the completion of operational transitions,we denote C（）as the operational condition observed just before a transition happens,and C（）as the operational condition just after the transition fi nishes,both of which are assumed to be at time t=. ｛B（n）:n≥0｝is a standard Brownian Motion(BM)process,and γ（γ ＞0）is its diffusion parameter,i.e.,γB（ n） ～N（ 0，γ2nh）.This term models the random error that cannot be attributed to the operational conditions.

In this paper,we consider an equi-distant conditionmonitoring situation,and assume that the cycle length h is so short that it only allows for one operational transition at most happening at the right end of each cycle.Therefore,given operational condition C（n）=k at the current sampling epoch tn=nh and C（n+1）=k′at the following sampling epoch tn+1=（n+1）h,the increment of the degradation signal during one operating cycle（ nh，（n+1）h］can be written as

where I（ C（n） ≠C（n+1））is the indicator function which equals to 1 if C（n）≠C（n+1）and 0 otherwise.

We assume that the rate of degradation μ（k）and the jump magnitude J（k，k′）can either be a deterministic value or a random variable,to include the possibility of randomness.While they are random,they are supposed to follow the normal law.μ（k）,J（k，k′）,and γB（t）are mutually independent.Thus,according to Eq.(6),the degradation increment Z（n+1）is also anormalvariable.Moreover,themeanandvarianceofZ（n+1）depend only on operational conditions C（n+1）and C（n）.

Thus,the model described by Eq.(6)can be simplified.We de fine Zk，k′（n+1）as the signal increment with parameters determined by C（n+1）=k′and C（n）=k,and de fine G（z|k，k′;Θk，k′）as the Probability Density Function(PDF)for the normal variable Zk，k′（n+1）given operational conditions C（n）=k and C（n+1）=k′.According to Eq.(6),G（z|k，k′;Θk，k′）is independent of time tnand only depends on operational conditions k and k′observed at adjacent sampling epochs.Thus,all possible G（z|k，k′;Θk，k′）form the set ofwith different combinations of k and k′.For M operational conditions,the set G has M2elements.Θk，k′is the vector of parameters in G（z|k，k′， Θk，k′）,determined by C（n）=k and C（n+1）=k′,which consists of mean ηk，k′and varianceare unknown and can be estimated by historical data.

Since the following RUL prediction approach is only interested in the distribution of the signal increment Z（n）,we do not have to estimate μ（k）,J（k，k′）,and γ one by one,but instead estimateandsimultaneously.Asandrepresent the mean and variance of the signal increment（n）for all n,we thus gather all signal increments between two adjacent sampling epochs in historical data and calculate their means and variances according to different combinations of k and k′(k，k′∈E).Supposing that we have L historical units,and for each unit l,we observen）at times...,and（n）at timeswhere ξlis the number of observations ofn）and ξl+1 is the number of observations of Ylk′（n）,then the least square estimates ofandare given by

2.3.Failure mechanism

In this work,we will consider a more general failure mechanism.Firstly,we still consider the most commonly used failure rule,referred to as threshold-crossing rule,to determine the failure time of the system.According to this rule,when the degradation signal reaches a predetermined threshold,the system is supposed to fail.Inspection is needed to discover the failure.Secondly,instead of using a uniform failure threshold for all operational conditions,as done in many RUL prediction approaches dealing with time-varying operational conditions (see Refs.1,8),we considerusing differentfailure thresholds according to different operational conditions.This failure mechanism is realistic and appropriate in many real applications,for example,a deteriorated system may not be able to function under harsh operational conditions,but it may work safely under mild conditions.Therefore,supposing that FLkis the failure threshold under operational condition k,we de fine the system threshold function as follows:

We then de fine the system end-of-life,the earliest time point at which the system threshold function evaluates to 1,as

The RUL of the system from a given current time point,na,is expressed as

3.RUL prediction using a semi-Markov decision process framework

RUL is de fined as the time length between the prediction starting point tna=nah and the ultimate failure,and it is estimated given all signal observations before tna.The prediction starting point is the sampling time for the latest observation of the degradation signal.In this paper,we will propose an SMDP-based algorithm for this RUL prediction problem,in one way trying to provide an alternative algorithm other than Monte Carlo-based methods for complicated RUL prediction problems and in the other way making an attempt to extend the applications of the SMDP.This algorithm only has to solve a limited number of linear equations to obtain the mean RUL.

The SMDP is a natural framework in which many policy decision models can be formulated.See the work of Tijms20for a general exposition of the SMDP framework.Our key of turning the whole RUL prediction problem into a problem that can be solved by an SMDP framework is the introduction of an appropriate‘‘policy”.This policy inspects a system's degradation signal at each sampling epoch,and lets the degradation signal start from the prediction starting time tnaand stop at the system failure time.After the system fails,this policy brings the degradation signal back to its‘‘initial”state at its prediction starting time tna,and lets the system operate again till its failure.Fig.1 shows a sample behavior of the degradation signal under this policy.In the view of this policy,the time length between the prediction starting time and the failure time equals to the total length of the system lifecycle CL(see Fig.1).Using this policy,the expected length of this lifecycle E（CL）corresponds exactly to the expected mean RUL of this system.

Costs are incurred as a consequence of policy decisions that are sequentially made when the degradation signal evolves over time.Two costs are incurred by this policy.One is the inspection cost CIat each sampling epoch,and the other is the cost of corrective maintenance CFwhich brings the system back to its‘‘initial”state.If we assign CI=0,then the expected total cost incurred in one lifecycle,denoted by E（ CC）,equals to CFwhen CF＞0.In an in finite time horizon,the average cost per unit time g of using this policy,by the renewaltheory,is equivalentto g=E（ CC）/E（CL） (see Ref.19).Therefore,if CI=0 and CF＞0,g=CF/E（CL）.

Now,to calculate E（CL）,the only remaining question is,how to calculate g without knowing the exact value of E（ CL）.Next,we will describe in detail how to calculate g using an SMDP framework.

Firstly,for any selected sensor s,the possible range of values of Yk（n）under operational condition k is required to be discretized into a finite set of states.Since the degradation signal under different operational conditions has different failure standards and initial values,we thus de fine［FLk，+∝)as the failure state Fkand HLkas the initial state of the degradation signal under operational condition k.FLkdenotes the failure threshold under operational condition k,which is usually given by expert knowledge.Then,we divide the continuous state space of［HLk，F(xiàn)Lk］,which is de fined as the healthy state,into Lkequidistant intervals with a constant length of Δk=（FLk-HLk）/Lk(k=1，2，...，M).Thus,the state space of the SMDP for situations under operational condition k can be denoted by Ωk=（Λk，E）,whererepresents the discretized states of Yk（n）,and E represents the operational condition under which Yk（n）is observed.Under such a discretization scheme,any value of Yk（n）can be assigned to a specific discrete state inFor example,if the degradation state at the prediction starting time tna=nah is found under operational condition ka,then Yka（na）belongs to the state ofwhereis the ceiling function.We use hereafter λka（na）to indicate the discrete state λkaat time tna=nah,and λka∈ Λk.

Secondly,the quantities in the SMDP should be determined,which are one-step transition probabilities of operational conditions, one-step transition probabilities of degradation,one-step expected sojourn times,and one-step expected costs.The two kinds of one-step transition probabilities characterize the probability law of the degradation signal's motion during one sampling interval.Among them,the ones of operational conditions are provided by Eq.(1),and the ones of degradation are derived using the degradation model described by Eq.(6).

For any tn=nh,if the current operational condition is k and the next operational condition is k′,the one-step transition probability that at next sampling time tn+1=（n+1）h,the value of Yk′（n+1） will be in the intervalgiven that the observed Yk（n） is in the intervalis calculated by

Similarly,the one-step transition probability that at next sampling time tn+1=（n+1）h,the value of Yk′（n+1）will be in the failure state Fk′given that the observed Yk（n）is in the interval［ λkΔk+HLk，（λk+1）Δk+HLk］(0≤ λk≤Lk),is calculated by

The one-step expected sojourn times characterize the expected times required by one-step transitions.In our situation,the expected time is always h,the length of the inspection interval.

Correspondingly,two different expected costs are required,which are where CIis the monitoring cost incurred at each sampling epoch,and CFis the cost of bringing the unit back to the state at the prediction starting time.Note that the two costs can be any arbitrary nonnegative numbers in this algorithm.For computational convenience,we set CIto 0 and CFto a positive integer.

Thirdly,once all of the above quantities are de fined,we are able to derive a system of linear equations which are used to calculate the long-run expected average cost per unit time g.We de fine a so-called relative value function of υ（ λka（na），ka）to formulate the relative cost in the in finite-horizon process when the system is in the state of（λka，ka）at its prediction starting time tna=nah.For t＞nah,we de fine another relative value function of υ（ λk，k）to formulate the relative cost when the system is currently in state（λk，k）,for all λkand k.

The relative value function of υ（ λka（na），ka）is determined by

The third term in Eq.(14)corresponds to the situation when the degradation signal will be in the healthy state at next sampling epoch,and the fourth term corresponds to the situation when the degradation signal will be in the failure state at next sampling epoch,for all operational conditions.Since we will certainly go through the next sampling right after the prediction starting time,the expected sojourn time should be h whatever λka（na）is.

Similarly,the relative value function υ（ λk，k）for each combination of λk∈ ｛1，2，...，Lk｝and k∈E is determined by

When λkis in the failure state Fk,the system fails,and will go back to the state of（λka，ka）.Therefore,the relative value function υ（ Fk，k）for each k∈E is determined by

Then,combining Eqs.(14)-(16)as a system of linear equations,we are able to calculate the long-run expected average cost per unit time g by solving this system of linear equations.We add one more equation to this system of linear equations to guarantee that the solution is unique,which is

Since only a single admissible action in each state is possible for the given policy,it is not necessary to formally apply the whole policy iteration procedure.We use the SMDP framework because it allows us to make efficient use of the linear equations in Step 1 of the policy iteration algorithm.In this algorithm,Lkis required to be set in advance.One can adopt a stopping rule similar to that in Ref.12to determine Lk.

After obtaining the long-run expected average cost per unit time g,E（RULna）can finally be estimated,which is given by

The procedure of the whole degradation modeling and RUL prediction is sketched in Fig.2.In this procedure,as shown in Eq.(14),the RUL prediction quite depends on the signal observation at the prediction starting time.However,it is quite risky to rely on only one observation,especially in a situation when the observation may have been contaminated by noise.Thus,in order to include more signal observations to reduce the effect of random errors,we take tna，tna-1，...，tna-prespectively as the prediction starting time to calculate the expected mean RUL.Let Es（RULna）denote the final result of the estimated RUL for sensor s,and we then have Es（RULna）calculated by

If we have r sensors for the prediction problem,then the final expected mean RUL is de fined as the average of all RUL prediction results obtained by each sensor as follows:

4.Case study and discussion

In this section,a case study is conducted on a NASA turbofan engine degradation dataset(see Ref.21)to illustrate our proposed RUL approach.

4.1.Simulated engine degradation data

The dataset we use is originally from the PHM08 Challenge Data21of the 2008 Prognostic Health Management(PHM)Conference Data Challenge Competition.This dataset provides time series sensory data of run-to-failure processes of a realistic large commercial turbofan engine,which is generated via a famous simulation process called C-MASS22.This dataset captures the engine's fault evolution until its failure,and records the multivariate time series sensory data for each run-to-failure process.Each time series collected at equidistant cycles represents the engine's run-to-failure process.We refer to one run-to-failure process as one unit hereafter.Each engine starts with different degrees of initial wear and manufacturing variation which are unknown to a user.The sensory data for each cycle of each unit includes the cycle index,the operational condition observed at the end of each cycle,and 21 values for 21 sensor measurements.

Fig.2 SMDP-based RUL prediction framework using single sensor.

Two datasets are available,a training dataset and a testing dataset,and each contains L=218 units.RUL prediction methods could be developed using the training dataset,and then they can be evaluated by the testing dataset.The sensory data for each run-to-failure process in the training dataset are complete from the start to the failure,but the ones in the testing dataset end sometime prior to complete failure.The objective of the competition is to predict the number of remaining operating cycles before complete failure for each unit in the testing dataset,i.e.,the number of operating cycles after the last cycle in which the engine will certainly continue to operate properly.In the following,we will use the training dataset to perform degradation modeling and the testing dataset to evaluate our RUL prediction approach.

In this Challenge Data,there are 3 operational settings that have a substantial effect on the engine performance.As listed in Table 1,the values of the 3 operational settings form 6 operational conditions in this dataset.Fig.3 shows an example(unit 1 in the training dataset)in which the 6 different operational conditions vary quite randomly over cycles.

Fig.3 also shows the corresponding time series sensory data of sensor 11 observed at sampling epoch(the end of each operating cycle)for training unit 1.It appears difficult to find an appropriate degradation model to directly describe this signal.However,when we extract sensor measurements under the same operational condition,for example,Fig.4 gives an example of sensor 11 measurements under operational condition 3 for training unit 1,we can clearly see that the degradation is highly correlated with the operational condition.Moreover,we plot in Fig.5 the initial value and the value at the failure time of the sensor 11 measurements for 218 training units,according to different operational conditions.It shows that the sensor measurements at failure times seem to gather in different zones according to different operational conditions.These two findings indicate that our degradation model in Section 2 might be able to describe the sensory data,and using a failure mechanism of different failure thresholds is very likely to be more appropriate for this dataset.

4.2.Selection of appropriate sensors

In this engine dataset,not all the sensors it provides are capable of capturing degradation characteristics.We therefore have to select at first effective sensors whose signals contain degradation information.There are 21 sensors to observe the health condition of the engine while it is operating.For eachsensor,we plot the time series sensory data under different operational conditions for the whole training population.Fig.6 gives a vivid picture of the time series sensory data of each sensor under operational condition 3,based on which we can divide the sensors into 3 groups(each shown in a separate row in Fig.6)according to the distinct characteristics of their sensory data.Note that the sensors in the third row are typical representatives for sensors 1,5,6,8,9,10,13,14,16,17,18,19,and 21.The sensor measurements in the first and second rows show a distinct upward or downward trend over time;therefore,their corresponding sensors are very likely to be appropriate sensors for degradation modeling.Other sensor measurements either show divergence or stay as one or several constant values,as the examples shown in the third row of Fig.6,so their corresponding sensors are not suitable for degradation modeling problems.Moreover,the sensors in the second row contain much more noise than those in the first row.Since preprocessing the sensory data is beyond the scope of our paper,we thus select the sensors in the first row S=｛2，3，4，11，15｝as the candidate sensors for our RUL predictions.

Table 1 Values of 3 operational settings for 6 different operational conditions in engine dataset.

Fig.3 Time series sensory data of sensor 11 and time series operational conditions for training unit 1.

Fig.4 Sensor 11 measurements under operational condition 3 for training unit 1.

Fig.5 Sensor 11 measurements at initial and failure times for 218 training units according to different operational conditions.

Fig.6 Signal observations under operational condition 3 for typical sensors and whole 218 training units.

4.3.Estimation of model parameters

Three groups of parameters are required to be estimated using the training dataset.The first group relates to the model of dynamic operational conditions.According to the method in Section 2.1,the results of^Pkk′for the engine dataset are listed in Table 2.We note that the results show that this DTMC is an irreducible ergodic Markov chain,and thus the limiting probabilities of the 6 operational conditions as well as their corresponding actual frequencies for the training dataset are shown in Table 2.They suggest that the DTMC model is very suitable to describe the dynamic process of the operational conditions for this engine dataset.

Table 2 Results of estimated transition probabilities of operational transitions for engine dataset.

Table 3 Parameter estimation results of degradation model for sensor 11.

The second group relates to the model of the degradation signal for each sensor.Section 3.2 presents the model and provides Eq.(7)to estimate model parameters.We have to estimate model parameters for 5 sensors in the sensor set S=｛2，3，4，11，15｝.Still taking sensor 11 as an illustrative example,we use the signal data of sensor 11 in the training dataset to calculate ηk，k′and σk，k′according to Eq.(7)for different k and k′(k，k′∈E).Results are listed in Table 3.Then we plot the probability plots of the signal increment in Fig.7.The figures show that no matter what operational conditions k and k′are,all signal increments conform very well to the normal law.Thus,the proposed degradation model is very suitable to describe the degradation signal of sensor 11.The same procedure is applied to othercandidate sensors aswell.Kolmogorov-Smirnov(K-S)goodness-of- fi t tests show that all P values are larger than 0.9,suggesting that the degradation model is very suitable for all candidate sensors.

The third group of parameters relates to the failure thresholds.The dataset does not specify a failure threshold for each degradation signal,so we have to establish it by the training dataset.According to the threshold-crossing failure rule,the failure time is supposed to be the first passage time of the failure threshold.Moreover,as analyzed in Fig.5,a conditionspecific failure mechanism is required.Therefore,we collect all the signal observations at the failure time for each historical failed unit and gather them in different groups according to a different operational condition k.We find that the observations in each group can be fitted to a normal distribution with mean μkand σk.Since we penalize late predictions more than early predictions(see Section 4.4 fordetails),we set FLk=μk-σkas the failure threshold for each operational condition k.The values of failure thresholds are listed in Table 4.

4.4.Evaluation of RUL prediction performance

Generally,the evaluation of RUL prediction approaches is based on functions of prediction errors which are de fined as

where L is the total number of units in the testing dataset(for this engine dataset,L=218).

We consider two traditional evaluation functions,the root squared error and the mean squared error,which evaluate how close the prediction is to the actual situation.They are de fined as

However,in many situations,to prevent a disaster resulted by failures is of the highest priority.Thus,an advance prediction is generally more desirable than a late prediction.That's why the of fi cial evaluation function in this Data Challenge adopted a specific scoring algorithm(see the work of Saxena et al.22for detail),in which heavier penalties on late predictions are stressed.The scoring algorithm is expressed as where φ1＞ φ2.The recommended setting of ｛φ1， φ2｝by this Data Challenge is｛φ1=13，φ2=10｝.

Fig.7 Probability plots of signal increments under different combinations of operational conditions k and k'for sensor 11.

Table 4 Estimated failure thresholds for all candidate sensors using threshold-crossing failure rule.

4.5.RUL prediction results and comparison

With all the estimated parameters,we are now able to estimate the RUL for each testing unit using our proposed SMDP-based algorithm.An appropriate value for Lkis required by this approach.See the work of Tang et al.12for a detailed description of the optimization scheme.Through the optimization scheme,we find that Lk=32 is large enough to ensure a convergence of prediction results.

Prediction is firstly performed based on each sensor,and then all the results from different sensors are combined according to Eq.(20)to give the final RUL prediction result for each testing unit.We evaluate prediction performance using the evaluation metrics de fined in Section 4.4.The evaluation results of the RUL prediction results based on each sensor and multiple sensors are listed in Table 5.We can see that the evaluation results for the finalRUL predictions are Score=3572,MSE=721,and RSE=398.We can also see that by combining the results from different sensors,the prediction performance is greatly improved.A comparison with the best obtained results in the PHM08 Data Challenge,which are the similarity-based prognostic method by Wang et al.23(Score=5636)and the neural network-based method by Heimes17(RSE=519.8)and Peel24(MSE=984),justify the accuracy of our approach.

Since our approach is a probabilistic approach,we also compare it with our probabilistic counterpart in literature,the wiener process-based method proposed by Le Son et al.25,which obtained the best results for this engine dataset among all existing probabilistic approaches in literature.In Ref.25,7 sensors were used and evaluation results were Score=5520,MSE=819,and RSE=423.Compared to their approach,our approach uses only 5 sensors,but shows a comparable performance in accuracy(re fl ected by RSE and MSE)and a better performance in Score.It indicates that our approach is more cost-efficient(more sensors require more investment)and obtains more early predictions.Therefore,our approach is more effective in maintenance and logistic optimization problems,which prefer early predictions than late ones.

Secondly,instead of fusing all sensory data to construct virtual degradation signals,we analyze the degradation signal of each sensor directly and predict the RUL based on a single sensor,so our degradation signal has physical meanings,and our approach is able to select sensors that are truly useful in describing the degradation process and predicting the RUL.For example,from the accuracy point of view(see MSE and RSE in Table 5),sensor 2 is most effective,then sensors 11 and 15,followed by sensors 4 and 3.Therefore,if only one sensor is allowed to implement the system,sensor 2 should be selected.For a situation of more sensors,one could use an optimization procedure called k-fold Cross Validation(CV)to find the optimal set.The procedure divides the original training dataset into k mutually exclusive subsets.Of the k subsets,one is used as the testing set,and the other k-1 subsets are put together as a training set.The procedure executes k times,with each of the k subsets used exactly once as a testing set.Then the CV error(which can be set as MSE,RSE,etc.)is computed as the average error over all k trials.With this procedure,we are able to find the optimal sensor set when the number of sensors is fixed.Moreover,we are able to find the preferred number of sensors.To reduce the number of sensors while retaining the effectiveness of condition monitoring is of great significance in real engineering systems.

Table 5 Evaluation results of RUL predictions for testing dataset.

5.Conclusions and future research

In this paper,we have proposed a novel RUL prediction approach using the SMDP framework.This approach deals with RUL prediction for a continuous-state degrading system under dynamic operational conditions.The effects of the operational conditions exerted on the degradation signal are quanti fi ed by different degradation rates and different signal jumps.In the equi-distant monitoring situation we have considered,the dynamic nature of the operational conditions is formed as a DTMC,and the degradation model is composed of degradation increments accumulated at each monitoring interval.Different failure standards according to different operational conditions are also considered,which complicates the problem butextendstheuse oftheproposed method in real applications.

Based on the above degradation model and failure mechanism,the RUL prediction problem is formed in an SMDP framework.An advantage of SMDP formulation is that the mean RUL can be obtained by solving a limited number of equations.The proposed RUL prediction approach is illustrated by a case study of a turbofan engine dataset,and its effectiveness is demonstrated by a comparison with results obtained by other existing approaches dealing with this dataset.

However,for this approach to fi t more real engineering practices,a few issues could be further analyzed and studied.First of all,as the number of operational conditions increases,it is quite necessary to find an appropriate approach to control the computational burden.Secondly,we only consider a DTMC to describe the dynamics of operational conditions,and extensions to models using a continuous-time Markov chain and the development of a corresponding RUL prediction algorithm are worth future researching.Moreover,it would also be interesting to use the proposed RUL prediction framework to investigate condition-based maintenance or predictive maintenance policies for degrading systems subject to dynamic operational conditions.

Acknowledgement

We would like to thank the anonymous reviewers for their valuable suggestions on this work and the National Natural science Foundation of China(No.71701008)for supporting this research.