12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12 Vancouver, Canada, July 12-15, 2015 1 Estimating Event Probabilities using Zero Failure Data Karl Breitung Researcher, TU Munich, Germany Marc A. Maes Professor, University of Calgary, Canada, and UGent, Belgium ABSTRACT: In a rather well publicized June 2013 CBC broadcast, the safety of a specific series of Canadian onshore gas pipeline joints was declared “absolute” by one proponent, since, historically, no incidents or failures had ever been reported for that site. An opponent then argued that the face value of this risk could never be zero, but “very small”. The objective of the present paper is to review just how small, and at which confidence level, one can sensibly consider the actual incidence rate to be. A comparison of the most popular approaches and a comprehensive test for consistency, point to the superiority of the Bayesian estimator together with a non-central posterior probability interval. 1. INTRODUCTION A common scenario in the context of data mining and explorative statistics of accidents/failures, is that the analyst runs into data segments or subsets that have zero reported incidents. Is it legitimate to construct, with a specified confidence, an upper limit pU for which we can claim that the “true” incident probability p will not exceed this value? A typical example concerns pipeline rupture and leak statistics (TSB, 2013): out of thousands of pipeline joints n a large majority of joints are observed to have no recorded incidents. There is often confusion about which confidence intervals apply in a case like this. Some analysts replace the zero incident case X = 0 by X = 1 because it is “easier” to analyze and because it is after all “conservative”. Others suggest that if, in the case of a binomial model, n consecutive trials have not resulted in any failure (X = 0), then we should prudently assume that the (n+1)-th will; subsequently this pessimistic estimate is used for prediction. 2. ESTIMATION FOR ZERO INCIDENTS One fundamental problem is related to the setting of the problem and to its formulation. In this paper we consider a binomial model originating from a setup where n units are monitored and incidents are observed within each unit. It also applies to the case where n sections of a continuous system such as a pipeline are tested and X are found to be faulty. The analysis for a Poisson model is entirely similar. Suppose we are to estimate for a sequence of n independent Bernoulli trials, the probability of failure p if X failures have been observed in this sequence. The (frequentist) estimator for p is: ?̂? =𝑋𝑛 (1) which suggests that, if no failures were observed, the estimator is zero. In a Bayesian setting, one starts from a prior distribution for the parameter p. Usually, the conjugate prior Beta distribution fpr is used with pdf: 𝑓pr(𝑝) =1𝐵(𝑎,𝑏)𝑝𝑎−1(1− 𝑝)𝑏−1 (2) where B(a,b) = (Г(a)Г(b))/Г(a+b) is the Beta function. The random variable with this pdf has a mean equal to a/(a+b) and a standard variance equal to ab/(a+b)2(a+b+1). Its shape is determined by the parameters a and b. Taking a = b = 1 gives a uniform distribution over the unit interval. Sometimes also the Jeffrey’s prior with 12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12 Vancouver, Canada, July 12-15, 2015 2 a = b = 0.5 is used, but here we will consider only the uniform prior, since it has some optimality properties as shown in the following. From this we then calculate the posterior using the likelihood of observations: ℓ(𝑋|𝑝) = �𝑛𝑋�𝑝𝑋(1− 𝑝)𝑛−𝑋 (3) yielding: 𝑓post(𝑝|𝑋,𝑛) = =1𝐵(𝑋+1,𝑛−𝑋+1)𝑝𝑋(1− 𝑝)𝑛−𝑋 (4) The Bayes estimator for a parameter is the mean of the posterior distribution fpost. Then the Bayes estimator for the probability p is in the case X = 0: ?̂? =1𝑛+2 (5) These results can be found in any standard text about Bayesian methods, e.g. in Press (1989), p. 40. So in the Bayesian setting the estimator is not equal to zero for X = 0 as in the frequentist case. The next issue concerns the derivation of sensible confidence intervals for the probability p. This poses also significant problems in the case X = 0. 3. THE CLOPPER-PEARSON CONFIDENCE INTERVALS The standard method for determining such intervals is described in detail in Clopper and Pearson (1934). They derive confidence intervals for the probability p in a frequentist setting. Unfortunately, this procedure does not give very satisfactory results for the case X = 0. In this case the confidence interval for the level α is in fact a confidence interval for the level α/2, i.e. it is too large. The original objective (Clopper and Pearson, 1934) is to calculate for a given confidence level α and an observed number of failures X as the lower bound pL for the confidence interval, the value pL for which: ∑ �𝑛𝑗� 𝑝𝐿𝑗(1− 𝑝𝐿)𝑛−𝑗 =(1−𝛼)2 𝑛𝑗=𝑋 (6) and as the upper bound pU the value for which: ∑ �𝑛𝑗� 𝑝𝑈𝑗 (1− 𝑝𝑈)𝑛−𝑗 =(1−𝛼)2 𝑋𝑗=0 (7) One has for the sums using integration by parts: ∑ �𝑛𝑗� 𝑝𝑗(1− 𝑝)𝑛−𝑗 = 𝑛𝑗=𝑋 = ∫𝑡𝑋−1(1−𝑡)𝑛−𝑋𝐵(𝑋,𝑛−𝑋+1)𝑑𝑑𝑝0 (8) as well as ∑ �𝑛𝑗� 𝑝𝑗(1− 𝑝)𝑛−𝑗 = 𝑋𝑗=0 = ∫(1−𝑡)𝑛−𝑋−1𝑡𝑋𝐵(𝑋+1,𝑛−𝑋)𝑑𝑑1𝑝 (9) So both bounds fulfilling the equations above can be found by inverting the incomplete Beta function with the respective parameters k and n – k + 1. The authors do not consider in the paper the case of X = 0 separately, but from the diagrams one sees that the lower bound is simply set to zero, since the sum on the left side of Eq. (6) is equal to unity, so the equation cannot be fulfilled. Therefore the confidence level is determined by the second equation only resulting in an interval with confidence level α/2. So its probability content is larger and it is longer. The Bayesian probability interval [pL, pU] for a specified “confidence” level α can be retrieved from the posterior pdf (4): Pr(𝑝𝐿 < 𝑝 < 𝑝𝑈|𝑋, 𝑛) = = ∫ 𝑓post𝑝𝑈𝑝𝐿(𝑝|𝑋, 𝑛)𝑑𝑝 = 𝛼 (10) For X = 0, the lower bound pL can be set to zero, without distorting the highest posterior density interval too much, so that pU can be found from inverting the incomplete beta pdf with parameters X + 1 and n – X + 1 (Box and Tiao, 1992): 12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12 Vancouver, Canada, July 12-15, 2015 3 ∫ 𝑓post𝑝𝑈0(𝑝|𝑋,𝑛)𝑑𝑝 = = ∫𝑝𝑋(1−𝑝)𝑛−𝑋𝐵(𝑋+1,𝑛−𝑋+1)𝑑𝑝 =𝑝𝑈0𝛼 (11) 4. COMPARISON OF METHODS In Pires and Amado (2008) a whole menagerie of about twenty different methods for calculating confidence intervals for the binomial proportions are presented. They are then compared with respect to criteria such as mean coverage probability and expected length. For a given method with number i, which produces the confidence interval [Li(j), Ui(j)] for j successes in n experiments, the coverage probability CP for given n and given p is given by the summation: CP(𝑝,𝑛, 𝑖) = = ∑ �𝑛𝑗�𝑗=0 𝑝𝑗(1− 𝑝)𝑛−𝑗𝐼[𝐿𝑖(𝑗),𝑈𝑖(𝑗)](𝑝) (12) with IA the indicator function for the interval A. For a fixed n, the mean coverage probability is found as: 𝔼�CP(𝑛, 𝑖)� = ∫ CP(𝑝,𝑛, 𝑖)𝑑𝑝10 (13) Then the expected length of an interval calculated with method i, given p and n is: Eℓ(𝑝,𝑛, 𝑖) = = ∑ �𝑛𝑗�𝑗=0 𝑝𝑗(1− 𝑝)𝑛−𝑗�𝑈𝑖(𝑗)− 𝐿𝑖(𝑗)� (14) Integrating over the parameter p gives then the overall expected length: 𝔼�Eℓ(𝑝,𝑛, 𝑖)� = ∫ Eℓ(𝑝,𝑛, 𝑖)10𝑑𝑝 = = ∑ �𝑛𝑗�𝑗=0 ∫ 𝑝𝑗(1− 𝑝)𝑛−𝑗�𝑈𝑖(𝑗)− 𝐿𝑖(𝑗)�10𝑑𝑝 (15) The coverage probabilities CP are shown for n = 40 and n = 400 in Pires and Amado (2008). These functions show quite erratic behavior as function of p. The twenty methods given in Pires and Amado (2008) are compared considering mean coverage probability, maximum and minimum coverage probability and then also mean, minimum and maximum length of the intervals. A method which is optimal for all possible criteria cannot be found, so mean coverage probability and mean length seem to be the most useful ones. The mean coverage should be as close as possible to the nominal coverage for which the intervals are constructed and the length of the intervals should be minimal. Consider now these two quality criteria for possible confidence intervals: 1. the mean coverage probability is to be as near as possible to the nominal coverage probability. 2. its length should be as short as possible. Due to the definition of the Bayesian confidence interval its mean coverage probability is always equal to the nominal. Considering the second criterion of mean length the Bayesian highest posterior density interval is always optimal, see p. 190 in Pires and Amado (2008). So the Bayesian approach using a posterior based on a uniform prior distribution gives “optimal” intervals if the criteria selected are the mean coverage probability and the mean length of the interval. In their conclusions Pires and Amado (2008) consider only central intervals thereby excluding the Bayesian intervals citing computational challenges related to non-central intervals. The exclusion of non-central intervals is not really justifiable, since nowadays the computational effort in computing such intervals has largely become irrelevant. For these non-central intervals no clear optimal methods are singled out. 5. NEED FOR CONSISTENT METHODS Consider once more the case where no failure has been observed till now. Let us assume two different methods for estimating the probability of such a failure have been applied, resulting in two different estimates of the probability: 12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12 Vancouver, Canada, July 12-15, 2015 4 ?̂?1 > ?̂?2 (16) From this follows that we can use for estimate of p: 𝑝 ≈ ?̂?1 (17) 𝑝 ≈ ?̂?2 (18) Subtracting the equations and dividing by ?̂?1 − ?̂?2 gives: 0 ≈ 1 (19) So one can derive from such inconsistent estimation methods false statements and ex falso sequitur quodlibet. How can we improve this situation? There have been proposals to improve the estimator in the case of zero observations by adding “virtual” observations. For example, if in n trials no failure has been observed, then we might assume that the first failure is just around the corner, i.e. it happens at the next trial. So the estimator for p is then: ?̂?1 =1𝑛+1=1𝑛−1𝑛(𝑛+1) (20) But for large n this is approximately equal to 1/n, so we here give almost the same probability weight to a case where no failures have been observed as to a case where one failure has been observed in n trials for which the estimator is: ?̂?2 =1𝑛 (21) If this is now plugged into a series system with k identical components, we would have as estimate for the failure Fk of the system in case 1: Pr(𝐹𝑘1)� ≈𝑘𝑛+1=𝑘𝑛−𝑘𝑛(𝑛+1) (22) whereas in the second case: Pr(𝐹𝑘2)� ≈𝑘𝑛 (23) which for large n is almost the same. So for small k we see that the probability of failure here is estimated as almost the same in spite of the fact that we have no failure observed in the first case. This leads to an unjustified and “conservative” shift of resources. The only reasonable recourse is to apply a fully Bayesian setting (Eqs. (2) to (5) above). Furthermore, one should also consider the actual context of the problem. For instance, if the data base is spatial in its structure – as in the case of pipeline incidents Canada-wide (TSB, 2013) – then a spatially hierarchical analysis can be performed where evidence of reported incidents is to a certain optimal extent “shared” over all assets resulting in a more relaxed treatment of “zero” incidents. 6. REFERENCES Box G.E.P. and Tiao G.C. (1992) “Bayesian Inference in Statistical Analysis”, Wiley, New York, 1992. Clopper C.J. and Pearson E.S. (1934) “The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial”, Biometrika, 26:404–423, 1934. Pires A. and Amado, C. (2008) “Interval Estimators for a Binomial Proportion: Comparison of Twenty Methods”, REVSTAT – Statistical Journal, 6(2):165–197, June 2008. Press S.J. (1989) “Bayesian Statistics: Principles, Models, and Applications”, Wiley, New York, 1989. TSB (2013) “Statistical Summary Pipeline Occurrences 2013”, Transportation Safety Board of Canada, http://www.tsb.gc.ca
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015) /
- Estimating event probabilities using zero failure data
Open Collections
International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015)
Estimating event probabilities using zero failure data Breitung, Karl; Maes, Marc A. Jul 31, 2015
pdf
Page Metadata
Item Metadata
Title | Estimating event probabilities using zero failure data |
Creator |
Breitung, Karl Maes, Marc A. |
Contributor | International Conference on Applications of Statistics and Probability (12th : 2015 : Vancouver, B.C.) |
Date Issued | 2015-07 |
Description | In a rather well publicized June 2013 CBC broadcast, the safety of a specific series of Canadian onshore gas pipeline joints was declared “absolute” by one proponent, since, historically, no incidents or failures had ever been reported for that site. An opponent then argued that the face value of this risk could never be zero, but “very small”. The objective of the present paper is to review just how small, and at which confidence level, one can sensibly consider the actual incidence rate to be. A comparison of the most popular approaches and a comprehensive test for consistency, point to the superiority of the Bayesian estimator together with a non-central posterior probability interval. |
Genre |
Conference Paper |
Type |
Text |
Language | eng |
Notes | This collection contains the proceedings of ICASP12, the 12th International Conference on Applications of Statistics and Probability in Civil Engineering held in Vancouver, Canada on July 12-15, 2015. Abstracts were peer-reviewed and authors of accepted abstracts were invited to submit full papers. Also full papers were peer reviewed. The editor for this collection is Professor Terje Haukaas, Department of Civil Engineering, UBC Vancouver. |
Date Available | 2015-05-15 |
Provider | Vancouver : University of British Columbia Library |
Rights | Attribution-NonCommercial-NoDerivs 2.5 Canada |
DOI | 10.14288/1.0076053 |
URI | http://hdl.handle.net/2429/53193 |
Affiliation |
Non UBC |
Citation | Haukaas, T. (Ed.) (2015). Proceedings of the 12th International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP12), Vancouver, Canada, July 12-15. |
Peer Review Status | Unreviewed |
Scholarly Level | Faculty |
Rights URI | http://creativecommons.org/licenses/by-nc-nd/2.5/ca/ |
Aggregated Source Repository | DSpace |
Download
- Media
- 53032-Paper_160_Maes.pdf [ 180.48kB ]
- Metadata
- JSON: 53032-1.0076053.json
- JSON-LD: 53032-1.0076053-ld.json
- RDF/XML (Pretty): 53032-1.0076053-rdf.xml
- RDF/JSON: 53032-1.0076053-rdf.json
- Turtle: 53032-1.0076053-turtle.txt
- N-Triples: 53032-1.0076053-rdf-ntriples.txt
- Original Record: 53032-1.0076053-source.json
- Full Text
- 53032-1.0076053-fulltext.txt
- Citation
- 53032-1.0076053.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.53032.1-0076053/manifest