International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015)

Risk-based decision making for deterioration processes using POMDP Nielsen, Jannie S.; Sørensen, John Dalsgaard Jul 31, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


53032-Paper_207_Nielsen.pdf [ 576.68kB ]
JSON: 53032-1.0076132.json
JSON-LD: 53032-1.0076132-ld.json
RDF/XML (Pretty): 53032-1.0076132-rdf.xml
RDF/JSON: 53032-1.0076132-rdf.json
Turtle: 53032-1.0076132-turtle.txt
N-Triples: 53032-1.0076132-rdf-ntriples.txt
Original Record: 53032-1.0076132-source.json
Full Text

Full Text

12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015Risk-based Decision Making for Deterioration Processes UsingPOMDPJannie S. NielsenPostdoc, Dept. of Civil Engineering, Aalborg University, Aalborg, DenmarkJohn D. SørensenProfessor, Dept. of Civil Engineering, Aalborg University, Aalborg, DenmarkABSTRACT: This paper proposes a method for risk-based decision making for maintenance of deterio-rating components, based on the partially observable Markov decision process (POMDP). Unlike mostmethods, the decision polices do not need to be stationary and can vary according to seasons and near theend of the lifetime. The approach is demonstrated through two examples, and the total expected costs aresimilar to those of another efficient method.1. INTRODUCTIONFor deterioration processes, maintenance can the-oretically be optimally planned using risk-basedmethods. Finding the optimal decisions in-volves solving a pre-posterior decision problemwith a large number of decisions, and the de-cision problem can, in principle, be solved us-ing either the normal or extensive analysis method(Raiffa and Schlaifer, 1961). Because the numberof branches in a traditional decision tree increasesexponentially with the number of time steps, itis generally not possible to calculate the expectedcosts for all branches.Previously, decision trees for maintenance plan-ning have been solved approximately by findingthe optimal stationary decision rules (Straub, 2004)(Nielsen, 2013). For maintenance of offshore struc-tures, inspections and repairs can only be per-formed during periods with relatively low windspeeds and wave heights, and the probability of in-spections and repairs being possible within a timeperiod will depend on the season. Therefore, thedecision maker could benefit from having time vari-ant decision policies that follow the seasons. Alsonear the end of the lifetime, decision policies willbe different, as preventive repairs should not bemade close to the end of lifetime.In this paper, an approach for solving these de-cision problems is considered, where the decisionpolicies do not need to be stationary. Here thecontinuous damage size is discretized, and the de-terioration processes are modeled using dynamicBayesian networks. Hereby, the approaches usedfor partially observable Markov decision processes(POMDP) can be used. The advantage of this ap-proach is that the computation time is only linearwith the number of time steps. The optimal deci-sion policies are found sequentially from the lastdecision for, in principle, all possible belief states.In reality, the belief state is represented by a vectorwith a sum equal to one, and the number of differ-ent belief states is infinite.In practice, the optimal decisions and expectedcosts can be found for a number of grid points, andthe results from the approximated belief state clos-est to the real one is used. The method has pre-viously been applied for a case where the damagestate can take three values: no damage, damage,and failure (Nielsen and Sørensen, 2012), and thebelief state could, therefore, simply be expressedby the probability of being in the damaged state, asit was assumed to be known whether or not failurehad occurred. The grid points for that case couldsimply be chosen as evenly distributed probabilitiesof damage between zero and one, and the accuracywas determined by the number of grid points.In this paper, the approach is extended to dete-rioration processes with more than three damage112th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015states. The approach is demonstrated for a main-tenance problem for offshore wind turbines.2. BAYESIAN PRE-POSTERIOR DECISION PROB-LEMFigure 1 shows the decision tree considered in thispaper. Information from condition monitoring andinspections is included, and decisions are made oninspections and repairs. Traditionally this type ofdecision problems is solved by using stationary de-cision rules. The decision on repair depends di-rectly on the most recent inspection outcome, andinspections are scheduled equidistant or when theprobability of failure given no detection at previousinspections exceeds a threshold. In both cases, theinspections are scheduled from the beginning, andextra information obtained during the lifetime is notconsidered.If condition monitoring is available, they can beincluded using this approach, by setting a thresholdfor the monitoring outcome for when inspectionsare made. However, only the most recent monitor-ing outcome is considered in this case, and in caseof uncertainties on the monitoring outcome, a betterand more informed decision could be made by in-cluding the history of monitoring outcomes. To doso, Bayesian updating needs to be done during thelifetime to estimate the current probability distribu-tion for the damage size. For this, discrete dynamicBayesian networks (DBN) can be applied, as theyenable computationally efficient Bayesian updating(Straub, 2009). Then a threshold for the probabilityof failure can be used as decision rule for when tomake inspections, simulations can be applied to de-termine expected costs and the optimal value of thedecision rules, and within the simulations a discreteBayesian network can be applied for updating ofprobabilities. This method is computationally ex-pensive, as time-consuming simulations are neededto find the optimal decision rule, and the decisionrules are stationary. If no time-invariant uncertain-ties are present, an alternative approach is to use amethod that exploits the Markovian assumption ofindependence between the future and past given thepresent.2.1. Markov decision model for deterioration pro-cessesA traditional Markov decision problem uses the factthat the optimal decision at a given time only de-pends on the current state of the component, not thehistory of damage development. If the componenthealth is directly observed at every time step, theoptimal decision for each time step can be found forall possible damage states sequentially from the lastdecision. Dynamic programming can be applied,such that the expected costs found for later timesteps are used when computing the optimal deci-sions for earlier time steps (Dasgupta et al., 2006).If the component health is not directly observed,but instead observed through an indicator, the prob-lem is a partially observable Markov decision pro-cess (POMDP). Here, the optimal decision at eachtime step only depends on the current belief statefor the component health. In other words, it de-pends only on the current probability distributionfor the damage size, as it summarizes the predictionfrom the model and all past observations. In prin-ciple, the optimal decision can then be found forall possible probability distributions for the dam-age size for each time step. However, in reality,there are infinitely many possible probability dis-tributions, so an approximation needs to be made.This can be done by finding the expected costs andoptimal decisions for a number of grid points, andthen interpolate between these grid points, whenexpected costs for other points are needed. To usethe approach for deterioration processes with (dis-cretized) continuous damage sizes, grid points needto be selected and a method to interpolate betweengrid points needs to be developed.In order to make a grid, the probability distribu-tion for the damage size is approximated by a 2-parameter Weibull distribution with scale parametera and shape parameter b, with cumulative distribu-tion function:FX(x) = 1− exp(−(xa)b)(1)The Weibull distribution is discretized and trun-cated before the failed state, as it is assumed to beknown if failure has occurred. The calculation grid212th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015inspectionno inspectionMonitoringDecision Ins. possibleNot possibleInspection resultrepairno repairDecision Rep. possibleNot possibleFailureRep. possibleNot possibleNo failureFigure 1: Decision tree for decisions on inspections and repairs and damage indications from condition moni-toring and inspections. At the dashed lines all branches continues as the ones illustrated. Repeated for each timestep in the made by doing this for a range of values of a andb.There are two approximations regarding the useof this approach. First, there is the variation be-tween the true probability distribution and the dis-cretized Weibull distribution closest to the true dis-tribution. Second, there is the approximation intro-duced by interpolation between grid points for thea and b parameters of the Weibull distribution.In order to find the Weibull distribution closestto the true distribution, a selection criterion needsto be set up. Possible choices include least squareestimates based on the distretized probability massfunction and least square estimates based the dis-cretized cumulative distribution function. The lat-ter has been used here, as differences in probabili-ties for nearby damage states are less critical thanfor damage states far from each other.A nonlinear optimizer can then be applied for es-timating the optimal values of a and b for any distri-bution. Thereafter, multidimensional linear or cu-bic interpolation can be used. For this application,the probability distribution for all grid points areknown, and an alternative interpolation method isto calculate the sum of the squares of the errors forall the distributions and select the distribution withthe lowest value. With this approach, the nearestdistribution is chosen and as such no interpolationis performed. For the same number of grid points,this method is less accurate, but it is much faster, asthe time-consuming nonlinear optimization is notneeded. Therefore, a denser grid can be used forthis method with same computation time, and it hasbeen used for the examples in this paper.In each time step, interpolation has been per-formed at two points. One after the condition moni-toring outcome is obtained, and one after correctiverepair. The outcome of the calculations are the de-cision policies for inspections and repairs for eachtime step. For inspections, the decision policies aregiven as function of the a and b values correspond-ing to the nearest Weibull distribution to the proba-bility distribution updated after condition monitor-ing. For preventive repairs, the decision policies aregiven as function of the inspection outcome as wellas the a and b values corresponding to the near-est Weibull distribution. Updating the probabilitydistribution for the damage size due to deteriora-tion and observations is performed using a discreteBayesian network approach.3. EXAMPLE 1The method is illustrated using a damage modelwith 10 damage states of equal size and constanttransition probability. The lower interval bound-aries are 0,1,2, . . . ,9, and the last state is the failedstate. This corresponds to linear damage growth.The lifetime is 20 years, and the mean time to fail-ure is 20 years. It is assumed that the damage sizecannot skip any state. Initially, the damage sizeis assumed to be in the first state with probabilityequal to one.3.1. ModelThe computation is run for a lifetime equal to 20years, and the step length is one month. This gives240 time steps in total. In the beginning of each312th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015time step, results from an online condition moni-toring system is obtained. The outcome dependson the damage size in the same way as for PoD(probability of detection) curves commonly usedin risk-based inspection planning. However, heremore outcomes are possible. The damage size caus-ing each monitoring outcome is assumed lognormaldistributed with parameters given in table 1, and fig-ure 2 shows the probability of obtaining each out-come as function of damage size. After the moni-toring outcome is obtained, a decision can be madeto make an inspection. Here a similar model isused, see table 2 and figure 3. After the inspectionoutcome is obtained, or if no inspection is made, adecision can be made to make a preventive repair.Then, the deterioration model is used to update thedamage size, and if failure happens during the timestep, a corrective repair is made.State Description Mean COV1 no alarm - -2 low alarm 2.0 1.03 high alarm 5.0 1.04 failure 9.0 0.0Table 1: Mean and coefficient of variation (COV) forthe damage sizes causing each monitoring outcome.Damage size0 2 4 6 8 10Probability of monitoring outcome00.511.51: No alarm2: Low alarm3: High alarm4: FailureFigure 2: Probability of each monitoring outcome asfunction of damage size.The costs are set relative to the costs of an in-spection, such that the expected costs of an inspec-tion is one, the expected costs of a preventive repairis 20, the expected costs of failure is 500, and theexpected costs of lost production per time step is100.State Description Mean COV1 no detection - -2 mild damage 2.0 1.03 some damage 4.0 0.84 significant damage 6.0 0.65 severe damage 8.0 0.46 failure 9.0 0.0Table 2: Mean and coefficient of variation (COV) forthe damage sizes causing each inspection outcome.Damage size0 2 4 6 8 10Probability of inspection outcome00.511.51: No detection2: Mild damage3: Some damage4: Significant damage5: Severe damage6: FailureFigure 3: Probability of each inspection outcome asfunction of damage size.Furthermore, the probability that an inspection orrepair is not possible within a time interval is in-cluded, and it can vary according to seasons. Foreach season, each lasting three months, a probabil-ity of the actions not being possible is defined. Ingeneral, there are stricter weather requirements formore complicated actions, so the probability thatcorrective repairs are not possible is larger than forpreventive repairs. And if preventive repairs arenot possible, neither are corrective. Therefore, theprobability that corrective repairs are not possibleduring a time step is provided conditioned that pre-ventive repairs are possible. Similarly, the proba-bility that preventive repairs are not possible is pro-vided conditioned that inspections are possible.The calculations are performed both for the casewithout and with seasons. Without seasons, it isassumed that inspections and repairs can alwaysbe made during the time step for which they areplanned. When seasons are included, inspectionsare still always assumed to be possible. Preventiverepairs are always possible during the summer half,but in the winter half there is a probability of 0.1412th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015a2 4 6 8 10b12345InspectNoinspectionNoinspectionInspection policy-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (1=no detection)-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (2= mild detection)-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (3= some detection)-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (4= significant detection)-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (5= severe detection)-20-15-10-5051015a2 4 6 8 10b12345 No repairRepair policy (6= failure)-20-15-10-5051015a2 4 6 8 10b12345 No repair RepairRepair policy (no inspection)-20-15-10-5051015Figure 4: Decision policies 10 years into the lifetime for Example 1. The color shows the difference in costs formaking the inspection/repair or not doing it. Positive values indicate that the optimal decision is to inspect/repair,and the black lines divide the regions with different optimal decisions. The repair policy is shown for each inspec-tion outcome and for no inspection.that it cannot be made during a time step. For cor-rective repairs, the probability that it cannot be doneduring a time step given that a preventive repair canbe made is 0.05 during summer half and 0.4 duringthe winter half.The range and spacing for the parameters a and bneed to be chosen based on a tradeoff between ac-curacy and computation time. The probability dis-tribution for the damage size was found for eachtime step for various combinations of observations,and the corresponding range of a and b values wasfound. Values of a are chosen in the range from0.25 to 11.5 with 0.05 distance between values. Forb the values are in the range from 0.5 to 5.5 withdistance 0.125. Additionally, the expected costs arefound for the failed case. This gives 9267 probabil-ity distributions in total for the damage size. Thecomputation time per time step was around 2 min-utes, and the total computation time for 240 timesteps was around 8 hours on an Intel Core i7 pro-cessor using parallel computing in Matlab.3.2. ResultsThe outcome of the computations is a set of deci-sion policies for each time step in the model. Fig-ure 4 shows an example for year 10, for the casewithout seasons. Not all policies are relevant forall values of a and b. For example, if an inspec-tion should not be made, it does not matter whatthe optimal repair decision is for each inspectionoutcome. Therefore, the policies can be summa-rized in a single figure as shown in figure 5. Asseasons are not included, policies for adjacent timesteps are very similar. However, near the end of thelifetime (20 years), the policies will change. Figure6 shows decision policies for year 18. As expected,damages should be larger before they are repairedcompared to year 10.When seasons are included in the model, the de-cision policies will generally vary during the year.Figure 7 shows the decision policies for inspectionsfor all months in year 10. During the summer half,inspections should be made at damages lower thanin the winter, such that repairs are less likely to bemade during the winter.To validate the efficiency of the found decisionpolicies, simulations are run where the found poli-cies are applied each time a decision is made.For comparison, simulations are also run for time-invariant decision policies, where the inspections512th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015Always repairNever repairInspect, repair if significant damage (4)Inspect, repair ifsevere damage (5)a2 4 6 8 10b0.511.522.533.544.555.5Figure 5: Summarized decision policies 10 years intothe lifetime for Example 1.Always repairNever repair Inspect, repair ifsignificant damage (4)Inspect, repair if severe damage (5)a2 4 6 8 10b0.511.522.533.544.555.5Figure 6: Summarized decision policies 18 years intothe lifetime for Example 1.a2 4 6 8 10b0.511.522.533.544.555.5inspectionNoInspectinspectionNo123456789101112Figure 7: Decision policies for inspections for allmonths in year 10 for Example 1. The first three andlast three months are considered winter months.are made when the probability of failure during thefollowing time step, which is updated using themonitoring outcome, is above a threshold value,and repairs are made when the inspection result isabove a threshold value. Both threshold values areoptimized using simulations. For both types of de-cision rules, Bayesian updating is performed duringsimulations using a DBN approach, and 100,000simulations are made for each case. Figure 8 showsthe expected costs for both cases. The two methodsare almost equally good, but the threshold approachgives slightly lower costs compared to the POMDPpolicies, both for the case with and without seasons.For comparison, the expected costs are 347 whenonly corrective maintenance is used.MethodPf POMDP Pf (season) POMDP (season)Total expected costs0510152025303540Inspection costsRepair costsFailure costsLost production costsFigure 8: Expected costs for POMDP and thresholdapproach (Pf) both with and without seasons for Exam-ple 1. The vertical black lines show the 95% confidenceintervals for the total costs.4. EXAMPLE 2For this example, the damage model is based on afracture mechanical model. The damage size (cracklength) a can be found based on the damage size inthe previous time step using the following expres-sion (Ditlevsen and Madsen, 2007):at =((1− m2)C∆Smpim/2∆n+a1−m/2t−1)(1−m/2)−1(2)Where ∆S is the stress range, ∆n is the number ofstress cycles, and m and C are empirical model pa-rameters. For this example, the time steps is onemonth, ∆S is assumed normal distributed with mean60 and standard deviation 10, ∆n is deterministic106, and m is deterministic with value 3.5. Theinitial value a0 is assumed exponential distributedwith mean value 0.2. The value of C is found bycalibration using Crude Monte Carlo simulations togive same mean time to failure as in Example 1. Avalue of C = e−33.5 was found using 100,000 simu-lations.612th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 20154.1. ModelNext, a DBN model for the damage developmentwas made following (Straub, 2009). As the damagemodel is exponential, the intervals for the damagesize have an exponentially increasing size. To findthe transition matrix, Monte Carlo simulations wereused. A good accuracy could be obtained using 80intervals for the damage size, but that would resultin long computation time (more than one week) forthe Markov model. Instead 30 intervals were used,even though it gave an overestimation of the prob-ability of failure. This was corrected by decreasingall probabilities below the diagonal by a constantfactor, and increasing the probabilities on the di-agonal to keep a total probability of one for eachinterval. The factor was chosen such that the prob-ability of failure after 20 years was equal to thevalue found from the original model using MonteCarlo simulations. Figure 9 shows the probabil-ity of failure as function of time for three cases:Monte Carlo simulations including 95% confidenceintervals, DBN model with 30 states, and the editedDBN model with 30 states.Time [years]0 5 10 15 20Probability of failure00. Carlo Simulations95 % confidence intervals30 state model - original30 state model - correctedFigure 9: Probability of failure as function of time forthe original model, DBN model with 30 states, andcorrected model with 30 states for Example 2.The range of a and b values was found in a sim-ilar way as in Example 1, and the range for a was0.05 to 10 with a step length 0.05, and for b therange was from 2.5 to 8 with a step length 0.125.In total, 9001 probability distributions including thedistribution for a failed component. Each time stephas a computation time of around 9 minutes, givinga total computation time of 36 hours.4.2. ResultsThe decision policies were found for the case withseasons, and figure 10 shows the summarized de-cision policies for year 10. Generally, inspectionsshould not be made, but repairs should be made atdistributions with lower scale parameters comparedto the linear model in Example 1. To validate theefficiency of the decision policies, simulations havebeen run as in Example 1 and the total expectedcosts are compared to the threshold approach in fig-ure 11. The two methods give almost the same totalexpected costs. For comparison, the expected costsfor corrective maintenance only is 682.Always repairNever repaira1 2 3 4 5 6 7 8 9 10b345678Figure 10: Summarized decision policies 10 years intothe lifetime for Example 2.MethodPf (season) POMDP (season)Total expected costs0510152025Inspection costsRepair costsFailure costsLost production costsFigure 11: Expected costs for POMDP and thresholdapproach (Pf) for Example 2. The vertical black linesshow the 95% confidence intervals for the total costs.5. DISCUSSIONThe examples show that the POMDP method is ableto give almost as good results as the efficient sim-ulation based threshold method. It was expected712th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015that it would give lower costs, especially, when sea-sons are included, as the POMDP method takes thatinto consideration and can have annual variations inthe decision policies unlike the threshold method.Higher costs of lost production and lower access-abilities during the winter could possibly make thePOMDP approach more beneficial.For the threshold method, only the probability offailure during the following time step is considered,when decisions are made. It has a direct relation-ship with the expected failure costs, but not with theexpected inspection outcomes and as such not theexpected costs to preventive repairs. The POMDPmethod considers the entire probability distribu-tion and, therefore, the relationships with both ex-pected failure and repair costs. If the probabilitydistribution could always be well approximated bya Weibull distribution and if proper interpolationwas performed, the POMDP method should givethe lowest costs. However, a 2-parameter Weibulldistribution does not always give a perfect fit, espe-cially when observations are included. A better fitcould be obtained by introducing a lower bound us-ing a 3-parameter Weibull distribution. If the com-putation time should still be limited, the number ofa and b values should be reduced to keep the sametotal number of probability distributions.The time-consuming part of the computation isto find the nearest grid point or, alternatively andeven more time-consuming, to make a nonlinear fitto a Weibull distribution. Therefore, the numberof times this is done will have a linear effect onthe computation time. The number of times the in-terpolation is done is the product of the number oftime steps, the number of grid points, and the num-ber of branches for each time step. Additionally,the time spent on each interpolation depends on thenumber of damage states and the number of gridpoints.A drawback of the POMDP model is the Marko-vian assumption, as time-invariant parameters arehard to include in the model. To do so, the gridpoints should be found for ’all possible’ joint dis-tributions for the damage size and a time-invariantmodel parameter. Even a relatively simple modelwith two parameters for each variable and a cor-relation would give a five-dimensional grid. Asthe computation time increases at least linear withthe number of grid points, it will probably be tootime-consuming. For the threshold approach, time-invariant parameters can easily be included in themodel.ACKNOWLEDGEMENTSThe work presented in this paper is part of theproject ”Reliability-based analysis applied for re-duction of cost of energy for offshore wind tur-bines” supported by the Danish Council for Strate-gic Research, grant no. 2104-08-0014. The finan-cial support is greatly appreciated.6. REFERENCESDasgupta, S., Papadimitriou, C., and Vazirani, U.(2006). Algorithms. McGraw-Hill.Ditlevsen, O. and Madsen, H. O. (2007). Structural Re-liability Methods. Department of Mechanical Engi-neering, Technical University of Denmark.Nielsen, J. S. (2013). “Risk-based operation and main-tenance of offshore wind turbines.” Ph.D. thesis, De-partment of Civil Engineering, Aalborg University.Nielsen, J. S. and Sørensen, J. D. (2012). “Mainte-nance optimization for offshore wind turbines usingPOMDP in Reliability and Optimization of StructuralSystems: Proceedings of the 16th working confer-ence on the international federation of informationprocessing (IFIP) working group 7.5 on reliabilityand optimization of structural systems. ed. A. Der Ki-ureghian, A. Hajian. American University of ArmeniaPress, Yerevan, Armenia, 2012. p. 175-182.Raiffa, H. and Schlaifer, R. (1961). Applied statisticaldecision theory. Harvard University.Straub, D. (2004). “Generic approaches to risk basedinspection planning for steel structures.” Ph.D. thesis,Swiss Federal Institute of Technology, ETH Zurich.Straub, D. (2009). “Stochastic modeling of deteriorationprocesses through dynamic bayesian networks.” Jour-nal of Engineering Mechanics, 135(10), 1089–1099.8


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items