International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015)

Uncertainty management of safety-critical systems : a solution to the back-propagation problem De Angelis, Marco; Patelli, Edoaro; Beer, Michael Jul 31, 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


53032-Paper_420_Angelis.pdf [ 222.79kB ]
JSON: 53032-1.0076201.json
JSON-LD: 53032-1.0076201-ld.json
RDF/XML (Pretty): 53032-1.0076201-rdf.xml
RDF/JSON: 53032-1.0076201-rdf.json
Turtle: 53032-1.0076201-turtle.txt
N-Triples: 53032-1.0076201-rdf-ntriples.txt
Original Record: 53032-1.0076201-source.json
Full Text

Full Text

12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015Uncertainty Management of Safety-Critical Systems: A Solution tothe Back-Propagation ProblemMarco de AngelisGraduate Student, Inst. for Risk and Uncertainty, School of Eng., Univ. of Liverpool,Liverpool, UKEdoardo PatelliLecturer, Inst. for Risk and Uncertainty, School of Eng., Univ. of Liverpool, Liverpool,UKMichael BeerProfessor, Inst. for Risk and Uncertainty, School of Eng., Univ. of Liverpool, Liverpool,UKABSTRACT: In many engineering applications, the assessment of reliability has to be done within alimited amount of information, which does not allow to use exact values for the distributional hyper-parameters. This is achieved defining probability boxes and assessing the reliability computing the failureprobability bounds. Probability boxes are often obtained from known probability distribution functionsrepresented by interval hyper-parameters. In the applications, not only it is of interest estimating the fail-ure probability bounds, but it is also required to identify the extreme realizations leading to the estimatedbounds. In this paper, we propose a strategy, based on the Kolmogorov-Smirnov test, to identify theparental distribution function that best fit the distribution of extreme realizations, obtained from the min-max propagation. From the results obtained comparing the strategy with a direct search, it has emergedthat the proposed method is generally applicable and efficient.1. INTRODUCTIONIn reliability assessment it is of interest comput-ing the effect of epistemic uncertainty on the failureprobability, (see e.g. Patelli et al. (2014), Roy andOberkampf (2010)) and making the least amount ofassumptions (see also Beer et al. (2013)). This re-quires the epistemic uncertainty to be propagatedthroughout the model and consequently quantifiedin terms of failure probability intervals. Uncer-tainty propagation can be performed by means ofdifferent strategies, nonetheless a general distinc-tion can be drawn between parametric (de Angeliset al. (2015), Zaffalon (2002)) and non-parametricapproaches (Alvarez (2006), Ferson et al. (2002),Kreinovich (1997)). The non-parametric proves tobe a more general approach because only boundsof the statistical input quantities are concerned andthere is no need to specify any parental probabilitydistributions. However, this approach comes withsome limitations, in fact, once the bounds of the in-terval probability are computed, it is not possibleto go back and identify the arguments in the inputspace corresponding to such bounds.In this paper we propose a numerical strategyto resolve the issue of back-tracking the failureprobability bounds in the input space, using sim-ulation methods and model updating procedure.The interval failure probability is obtained fromthe Dempster-Shafer (D-S) structure of the outputquantity of interest (Dempster (1967)). Any levelof the D-S structure identifies a minimum and amaximum of the quantity of interest produced bydifferent searching domains. The expected value ofall the minima (maxima) of the D-S structure repre-sents the lower (upper) bound of the expected valueof the quantity of interest. To any minimum (max-112th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015imum) corresponds an argument minimum (maxi-mum) in the input space. Therefore, by gatheringall the minima (maxima), and track these minima(maxima) back to the input space, it is possible toconstruct a cumulative distribution function (CDF)of the corresponding argument minima (maxima).Equivalently, but this time selecting the right com-bination of extrema, realizations in the input spacecorresponding to the lower and upper bounds of thefailure probability can be identified. These realiza-tions are then gathered to make up a cumulativedistribution function for each bound of the failureprobability, and subsequently interpreted, for exam-ple, by means of model updating techniques. Here,model updating is used to identify the best probabil-ity function that fits the two obtained CDFs corre-sponding to the lower and upper bounds of the fail-ure probability. Eventually, a (parental) probabilitydistribution can be chosen to best fit the resultingCDFs.2. RELIABILITY ASSESSMENT WITHPROBABILITY BOXES2.1. Brief introduction to probability boxesProbability boxes (p-boxes) extend the definition ofreliability to an interval of possible alternatives, en-closed by a lower and a upper bound. In reliabilityand risk assessment, p-boxes are invoked to repre-sent what in literature is referred to as uncertainty ofType III, which include both aleatory and epistemicuncertainties. Let F and F be non-decreasing func-tion from the real line R into [0,1] and F ≤ F for allx∈R. A p-box is the set of all non-decreasing func-tions F : R→ [0,1], such that F(x)≤ F(x)≤ F(x).2.2. P-box convolution by means of Monte CarlosimulationA Dempster-Shafer structure can be seen as the”discrete” equivalent of a p-box and it is key forthe reliability assessment of systems. The input-output convolution of p-boxes, in practice, is per-formed by using Monte Carlo simulation tech-niques (Kreinovich et al. (1991)). Reliability as-sessment is, therefore, performed by i) sampling theequivalent D-S structure of the p-boxes, ii) obtain-ing the output D-S structure, iii) and ultimately es-timating the failure probability bounds.2.3. Failure probability upper and lower boundsLet G : Rn → R be the system performance func-tion, θ ∈ Rn be a vector of p-boxes, and ΩF bethe domain of unacceptable states (or failure do-main), such that ΩF = {θ : G (θ)≤ 0}. The sys-tem performance is evaluated as g = G (θ). Eachfocal element, θ {s}, of the D-S structure is propa-gated throughout the system, and the correspondingimage is obtained asG (θ {s}) = [g, g]{s}; (1)where,g = minθ∈θ{s}G (θ); g = maxθ∈θ{s}G (θ). (2)The propagation of individual focal elements leadsto the failure probability bounds, obtained using theplausibility and belief function aspF = limNs→∞Ns∑G (θ{s})∩ΩFm(G (θ {s})); (3)pF = limNs→∞Ns∑G (θ{s})⊆ΩFm(G (θ {s})); (4)where, m is the mass associated to each focal ele-ment of the D-S structure.2.4. Reliability assessment with the parametricapproachThe failure probability bounds are obtained bysearching among all those distribution functions,which mean values and standard deviations are in-cluded in the intervals µ and σ . The problem canbe alternatively formulated aspF = minµ,σpF(µ,σ); pF = maxµ,σpF(µ,σ). (5)Using this approach, the solution is included in thep-box bounding CDFs (see e.g. Figures 1 and 2),and it also belongs to one of the distribution func-tions defined by the interval hyper-parameters (seee.g. Table 1). In case where the performance func-tion is a black-box, the approach is driven by anoptimization procedure that solves the problem ofEq. 5 numerically.212th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 20152.5. Reliability assessment with the non-parametric approachAny distribution function contained within thebounding CDFs, even not belonging to any parentaldistribution model, can be considered for the prob-lem solution. This implies that the failure proba-bility bounds obtained using this approach, are al-ways wider than those obtained from the paramet-ric approach, because a greater set of candidates issearched for. The failure probability bounds are ob-tained by computing plausibility and belief of theoutput D-S structure as shown in Eqs. 3 and 4.One major limitation of the non-parametric ap-proach, despite its efficiency, is the difficulty inidentifying the input distribution functions that areresponsible for the failure probability bounds. Thisissue is also known in literature as the tracking (orback-propagation) problem. Here we propose astrategy to tackle this issue.2.6. Solution to the tracking problemIn many applications, it is of interest identifying thedistribution functions that lead to the failure proba-bility bounds. Thus, a numerical strategy is neededto characterize the distribution model that best rep-resent the extreme realizations. Here, we proposea strategy, based on the Kolmogorov-Smirnov test,that identifies the function that best-fit the distribu-tions of extrema (Kolmogoroff (1941)). The prob-lem is formulated asminµ,σsupx|FNs(x)−F(x;µ,σ)| ; (6)where, supx |FNs(x)−F(x;µ,σ)|, is the K-S statis-tic, which represents a measure of similarity be-tween the CDF obtained with the sample setθ {s}min(max) and the distribution function F(x;µ,σ).Solution to the problem is the pair of hyper-parameters (µ∗,σ∗) that minimizes the K-S statis-tic for each input p-box.3. NUMERICAL EXAMPLEThe example is formulated to show limitations andadvantages of using the two approaches. The fol-lowing performance function will be consideredthroughout this section asg = x2 y+ ex; (7)−2 0 2 4 6 800.  CDFUCDFLµL, σLµL, σUµU, σLµU, σUFigure 1: P-box bounding normal CDFs for x−10 −8 −6 −4 −2 0 2 4 600.  CDFUCDFLµL, σLµL, σUµU, σLµU, σUFigure 2: P-box bounding normal CDFs for ywhere, x and y are p-boxes obtained from normaldistribution functions, which parameters are shownin Tables 1. Let the failure event be defined by thefailure region ΩF = {x,y : G (x,y)≤ 0}, then thefailure probability is expressed as the intervalpF = P[g≤ 0]. (8)The p-boxes defined in Table 1 are represented inFigures 1 and 2 in terms of bounding CDFs.The aim of this example is to identify the failureprobability bounds and the corresponding realiza-tions in the input space, i.e. those CDFs that yieldthe minimum and maximum failure probability.Depending on how the input space of candidatesolutions is searched for, the solution may be signif-312th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015P-box µ σ Distributionx [1, 5] [0.1, 0.6] Normaly [−2, −0.5] [0.6, 2] NormalTable 1: Mean values and standard deviations for thedefinition of the p-box boundsFigure 3: Limit state surface and box of mean valuesicantly different. In the next sections two differentapproaches of searching in the input space of can-didate solutions are presented.3.1. The parametric approachIn this simple case, the optimization can be reducedto a one-dimensional search. In fact, the function ofEq. 3 is monotonically increasing in y, which per-mits to discharge µyfrom the list of candidates, asµyand µy corresponds to pF and pF , respectively.Also, the standard deviations can be taken out of theoptimization as only four candidate solutions can beidentified, which correspond to the four corners ofthe domain σ . This becomes more evident if welook to the limit sate surface and the box of candi-date mean values, as shown in Figure 3. The opti-mization is, therefore, reduced to a search along thethick (upper and lower) edges represented in Fig-ure 3. Given the shape of the limit state surface,we expect the maximum failure probability to be lo-cated somewhere near the peak of the limit sate sur-face. Minimum and maximum failure probabilitiesare obtained on the upper and lower edges of the1 1.5 2 2.5 3 3.5 4 4.5µxp F max pFFigure 4: Failure probability values obtained from theoptimization along the lower µx edge1 1.5 2 2.5 3 3.5 4 4.5 53.544.555.566.577.58 x 10−3µxp Fmin pFFigure 5: Failure probability values obtained from theoptimization along the upper µx edgeµx domain respectively, populating the space with1000 realization. On each realization, the failureprobability is estimated using MC simulation with105 samples. Results from the edge optimizationare shown in Figure 4, where it is shown that themaximum failure probability (unlike the minimum)is attained within the edge, thus not at the cornersof the domain. Figure 5 shows that the minimumfailure probability is held at the right endpoints ofthe domain, i.e. for µx = 5. The argument optimaand corresponding failure probability extrema arereported in Table 2.412th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015min pF max pFP-box 5.30 10−9 0.493x (µx)· = 5.0 (µx)· = 3.49x (σx)· = 0.1 (σx)· = 0.1y (µy)· = -0.5 (µy)· = -2.0y (σy)· = 0.6 (σy)· = 2.0Table 2: Failure probability bounds and correspondingextreme normal distributions3.2. The non-parametric approachThe failure probability bounds are obtained by gen-erating a great number of samples, from the p-boxes, and subsequently constructing the associ-ated D-S structures of the response. The procedurefor constructing the D-S structure of the response isbriefly summarized as,1. draw a uniform random number, α{s}, for eachp-box, between 0 and 1;2. get the sample endpoints x{s} using the inversebounding CDFs, F−1as;x{s} = F−1(α{s}); x{s} = F−1(α{s}); (9)3. identify minimum, r{s}, and maximum re-sponse, r{s}, within the search domain, x{s}.This step is also referred to as min-max prop-agation;4. repeat the above steps for s from 1 to Ns, i.e.loop over the number of samples Ns;5. collect samples and corresponding responseextrema.Once the D-S structure of the response is obtained,the failure probability bounds are obtained from theD-S plausibility and belief aspF = limNs→∞1NsNs∑s=1I [g{s} < 0]; (10)pF = limNs→∞1NsNs∑s=1I [g{s} < 0]; (11)where, I : R → {0,1} is the indicator function.Most of the attention, in the above procedure, isusually given to the min-max propagation step. Infact, this can be troublesome, especially if the re-sponse is the output of a black-box model, thus thepropagation is done by invoking global optimiza-tion algorithms (such as evolutionary or stochasticalgorithms). However, in this case, solution to thepropagation task can be found analytically, as theperformance function is given explicit mathemati-cal expression.3.2.1. The min-max propagationThe performance function of Eq. 3 is monoton-ically increasing with respect to y, which is agreat advantage as it excludes the presence of rel-ative minima and maxima. Moreover, it impliesthat, for every value of x, as the variable y de-creases/increases so does the performance function.This leads to the following relationshipsg = x2∗ y+ ex∗; g = x∗2 y+ ex∗; (12)where x∗ and x∗ are yet to be determined. On theother side, the performance function is not mono-tonic with respect to x. The sign of the first andsecond derivatives of g, says that the function ismonotonically increasing with respect to x only inthe portion where y ∈ [−1.37,0]. Whereas, for y ∈(−∞,−1.37)∪ (0,∞) the function may have a min-imum or maximum. Within the latter portion of do-main, the minimum/maximum is identified solvingfor x the partial derivative ∂g/∂x, and subsequentlychecking if the obtained value is smaller/greaterthan the values at the endpoints x and x.3.2.2. Solution to the back-propagation problemThe problem is addressed collecting all those real-izations in the input space that correspond to theresponse extrema. It is, thus, crucial solving themin-max propagation problem by keeping track,back to the input space, of all the minima and max-ima. These are also referred to as extreme realiza-tions. The failure probability bounds are computedby means of Eqs. 10 and 11 using 105 MC samples.pF = [pF , pF ] = [0, 0.754].512th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015−6 −4 −2 0 2 4 600.  non−parametricparametricFigure 6: CDFs from the x p-box of extreme realiza-tions leading to the failure probability upper bound−8 −6 −4 −2 0 2 400.  non−parametricparametricBounding CDFFigure 7: CDFs from the y p-box of extreme realiza-tions leading to the failure probability upper boundThe strategy intends to use the distributions ofminima and maxima as probability models corre-sponding to the upper and lower bounds of the fail-ure probability, respectively. In Figures 6 and 7 thedistribution of minima, corresponding to the upperfailure probability, are compared with the extremenormal distributions of Table 2 for the maximumfailure probability, obtained by solving the opti-mization problem. Figure 7 shows quite clearlythat the normal distribution obtained from the para-metric approach, i.e. corresponding to the maxi-mum failure probability, fits quite well the distribu-tion of minima obtained from the non-parametricmin p∗F max p∗FP-box 1.8 10−5 0.468x (µx)· = 4.97 (µx)· = 2.43x (σx)· = 0.23 (σx)· = 0.60y (µy)· = -0.51 (µy)· = -1.98y (σy)· = 1.10 (σy)· = 1.07Table 3: Failure probability bounds and correspondingextreme normal distributions obtained with the non-parametric approachapproach. Figure 6 also shows a good fit, although,this time, it is clear that the normal distribution doesnot represent the best fit.The solution to the back propagation prob-lem can be found by selecting in the spaceof parental (normal) distribution functions, thosehyper-parameters corresponding to the min/maxfailure probability. Within the non-parametricapproach, this can be done searching for theparental distribution functions that provides thebest fit to the collected distributions of minimaand maxima. Here, the normal distribution thatbest-fit the extreme realizations is obtained usingthe Kolmogorov-Smirnov test, by minimizing thestatistic (k-s distance) DNs = supx |FNs(x)−F(x)|.The results from the the k-s distance minimizationare shown in Table 3. Figures 8 and 9 show the xand y extreme normal distributions obtained for thefailure probability upper bound. It is interesting tosee that the two normal distribution functions of ex-treme values obtained using the two approaches donot differ much. However, the failure probabilitybounds obtained using the proposed strategy, as itcan also be seen in Table 3, are not the optimal ones,as they are enclosed in the ones obtained using theparametric approach,which are shown in Table 2.Note, from Figure 9 that the extreme realizationsof p-box y are distributed as the upper CDF as themodel is monotonic with respect to this variable.3.3. Final remarksFrom the analysis of the extreme realizations withboth parametric and non-parametric approaches wemay conclude that– if the response is monotonic with respect to a612th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015−4 −2 0 2 400.  extreme realizationsnormal best−fitFigure 8: Failure probability upper bound: the normaldistribution, from the x p-box, that best-fit the extremerealizations is obtained with a k-s distance of 0.11−8 −6 −4 −2 0 2 400.  extreme realizationsnormal best−fitFigure 9: Failure probability upper bound: the normaldistribution, from the y p-box, that best-fit the extremerealizations is obtained with a k-s distance of 0.15p-box, the failure probability bounds are ob-tained from the bounding CDFs of that p-box,– if the response is not monotonic with respectto a p-box, the distribution function of the ex-treme realizations is enclosed in the boundingCDFs of that p-box and may have a compli-cated form,– in general, the reconstructed CDF of the ex-treme realizations is not distributed as theparental model of probability– if the response is monotonic with respect to allp-boxes and if, for every p-box, the boundingCDFs are made of only two distribution func-tions (such as in the Beta model), the solutionfrom the two approaches coincides.4. CONCLUSIONSIn many engineering applications the assessmentof reliability requires the consideration of uncer-tainty in the form of probability boxes. Very often,probability boxes are defined using known proba-bility distribution functions represented by intervalhyper-parameters. In these cases, it is of interest notonly estimating the bounds on the output statisti-cal quantity of interest, such as the failure probabil-ity, but it is also required to identify which extremerealizations led to the estimated bounds. While insome cases it may be sufficient just knowing whatthe mass function of these realizations is, in othercases it may be necessary to know what is the clos-est distribution function, from the underlying modelof probability. In this paper, we have proposed astrategy, based on the Kolmogorov-Smirnov test,to identify the parental distribution function that isclosest to the distribution of extreme realizations.The strategy collects the realizations from the min-max propagation, and search, in the space of feasi-ble hyper-parameters, for the distribution functionthat best-fit the collected data. From the results ob-tained comparing the strategy with a direct search,performed by means of the parametric approach, ithas emerged that the proposed method works welland shows also to be quite efficient. However, theaccuracy of the strategy might not be satisfactoryfor the lower bound of the failure probability, which712th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015is different from the optimal value by orders ofmagnitude. An improvement in this direction canbe sought, for example, putting more emphasis onthe tails of the fitted distributions. Also, as onemore future research direction, it would be interest-ing to see how choosing more than one probabilitymodel at a time can increase the confidence of theestimation and lead to a better representation of theextreme realizations.5. REFERENCESAlvarez, D. A. (2006). “On the calculation of thebounds of probability of events using infinite randomsets.” International journal of approximate reasoning,43(3), 241–267.Beer, M., Ferson, S., and Kreinovich, V. (2013). “Impre-cise probabilities in engineering analyses.” Mechani-cal systems and signal processing, 37(1), 4– Angelis, M., Patelli, E., and Beer, M. (2015). “Ad-vanced line sampling for efficient robust reliabilityanalysis.” Structural Safety, 52, 170–182.Dempster, A. P. (1967). “Upper and lower probabili-ties induced by a multivalued mapping.” The annalsof mathematical statistics, 325–339.Ferson, S., Kreinovich, V., Ginzburg, L., Myers, D. S.,and Sentz, K. (2002). Constructing probability boxesand Dempster-Shafer structures, Vol. 835. Sandia Na-tional Laboratories.Kolmogoroff, A. (1941). “Confidence limits for an un-known distribution function.” The Annals of Mathe-matical Statistics, 12(4), 461–463.Kreinovich, V. (1997). “Random sets unify, explain, andaid known uncertainty methods in expert systems.”Random Sets, Springer, 321–345.Kreinovich, V. Y., Bernat, A., Borrett, W., Mariscal, Y.,and Villa, E. (1991). “Monte-carlo methods makedempster-shafer formalism feasible.Patelli, E., Alvarez, D. A., Broggi, M., and Angelis,M. d. (2014). “Uncertainty management in multidis-ciplinary design of critical safety systems.” Journal ofAerospace Information Systems, 1–30.Roy, C. J. and Oberkampf, W. L. (2010). “A com-plete framework for verification, validation, and un-certainty quantification in scientific computing.” 48thAIAA Aerospace Sciences Meeting Including the NewHorizons Forum and Aerospace Exposition, 4–7.Zaffalon, M. (2002). “The naive credal classifier.” Jour-nal of statistical planning and inference, 105(1), 5–21.8


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items