12th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015Building Probability of Detection Curves via MetamodelsThomas BrownePhD Student, EDF R&D, Chatou, FranceLoïc Le GratietResearch Engineer, EDF R&D, Chatou, FranceGéraud BlatmanResearch Engineer, EDF R&D, Moret sur Loing, FranceSara CordeiroEngineer, EDF CEIDRE, Saint Denis, FranceBenjamin GoursaudResearch Engineer, EDF R&D, Clamart, FranceBertrand IoossSenior Researcher, EDF R&D, Chatou, FranceLéa MauriceEngineer, EDF CEIDRE, Saint Denis, FranceABSTRACT: Probability of Detection (POD) curves is a standard tool in several industries to evaluatethe performance of Non Destructive Testing (NDT) procedures. However, the classical methods for PODdetermination rely on strong statistical assumptions (linearity, residuals normality and homoscedasticity).In the context of numerical POD estimation (with data coming from numerical simulations of the system),we study classic and novel model-based approaches. Applications are performed on Eddy Current NonDestructive Examination numerical data.1. INTRODUCTIONIn several industries (as in aeronautics), the proba-bility of Detection (POD) curve is a standard tool toevaluate the performance of Non Destructive Test-ing (NDT) procedures (Gandosi and Annis, 2010;MIL-HDBK-1823A, 2009). The goal is to assessthe quantification of inspection capability for thedetection of harmful flaws for the inspected struc-ture. For the French company of electricity (EDF),the potentialities of this tool are studied in the con-text of the Eddy Current Non Destructive Examina-tion in order to ensure integrity of steam generatorstubes (Maurice et al., 2012).However, high costs of the implementation ofexperimental POD campaigns combined with con-tinuous increase in the complexity of configurationmake them sometimes unaffordable. To overcomethis problem, it is possible to resort to numericalsimulation of NDT process.In this work, we focus on the examination underwear anti-vibration bars (AVB) of steam generatortubes with simulations performed by the computercode Code_Carmel3D (developed by EDF R&D).The construction of the numerical model requiresa specific mesh, as displayed in Figure 1. In thefollowing experiments the input set was picked onlyin order to test the numerical models. It has notbeen validated.The determination of this “numerical POD” isbased on a four-step approach:1. Identify the set of parameters that significantlyaffect the NDT signal;112th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015Figure 1. Illustration of the mesh in the numericalmodel of NDT simulation.2. Attribute a specific probability distribution toeach of these parameters (for instance from ex-pert judgment);3. Propagate the input parameters uncertaintiesthrough the NDT numerical model;4. Build the POD curve from standard ap-proaches like the so-called Berens method.This approach is closely related to the genericuncertainty management methodology in numeri-cal simulation as explained in de Rocquigny et al.(2008) and Pasanisi and Dutfoy (2012). Severalstatistical tools based on numerical design of ex-periments, uncertainty propagation efficient algo-rithms and metamodeling concepts will then be use-ful (Fang et al., 2006).The model parameterization and the design ofnumerical experiments (Code_Carmel3D computa-tion) are explained in the following section. Thethird section introduces the three POD curves de-termination methods: the Berens method (basedon a linear regression model), a binomial-Berensmethod and a method based on the Gaussian pro-cess surrogate model. A conclusion section synthe-sizes the work and introduces our prospects.2. MODEL PARAMETERIZATION ANDNUMERICAL DESIGN2.1. Influent parameters and associated randomdistributions definitionBy relying on both experts reports and data simu-lations, the sample of the Influent Parameters (IP)which can have an impact on the code outputs cho-sen for this faisibility study are:• E ∼ N (aE ,bE) : pipe thickness (mm) basedon data got from 5000 pipes,• h1 ∼ U [ah1,bh1 ] : first flaw height (mm),• h2 ∼ U [ah2,bh2 ] : second flaw height (mm),• P1 ∼ U [aP1,bP1] : first flaw depth (mm),• P2 ∼ U [aP2,bP2] : second flaw depth (mm),• ebav1 ∼U [−P1+aebav1 ,bebav1 ] : length of thegap between the AVB and the first flaw (mm),• ebav2 ∼ U [−P2 + aebav2 ,bebav2 ] : length ofthe gap between the AVB and the second flaw(mm).As displayed in Figure 2, we consider the occur-rence of one flaw on each side of the pipe due toAVB. To take this eventuality into account in thecomputattions, 50% of the experiments are mod-elized with one flaw, and 50% with two flaws.Figure 2. Illustration of the considered inputs.2.2. Definition of the design of experimentsIn order to build a simplified model (i.e. a surrogatemodel) that estimates the output of interest Pro jYrelated to the IP of the system, it is needed to evalu-ate Code_Carmel3D on some points in IP set. Thisdataset, called design of experiments, has to be de-fined at the very beginning of the study, which is212th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015to say before any numerical simulation. A clas-sic method consists in building the design of ex-periments by picking completely randomly differ-ent points of the IP set (Monte Carlo simulationstype). However from time to time it leads to adesign which does not properly "fill-in" the IP set(Fang et al., 2006): the idea is to spread the numer-ical simulations all over the IP set so no big subsetis left "unknown". To this effect it is more relevantto choose the values according to a deterministicrule, such as Quasi-Monte Carlo method. Indeed,for a size of design N, it is proved that this methodoften happens to be more precise than the clas-sic Monte-Carlo method (Morokoff and Caflisch,1995). Given the available computing time, a de-sign of experiments of size 100 is created.3. METHODS OF POD CURVES ESTIMA-TIONIn this framework, one wants to build the PODcurve as a function of its most influent parame-ter: a := max(P1,P2). By using the computer codeCode_Carmel3D, one focuses on the output Pro jYwhich is a projection of the simulated signal wewould get after NDT process. The other inputs areseen as random variables, which makes Pro jY itselfan other random variable. The effects of all the IPare displayed in Figure 3. The bold values are thecorrelation coefficients between the output Pro jYand the corresponding IP. Strong influences of P1and P2 on Pro jY are detected.Given a threshold s > 0, a flaw is consideredto be detected when Pro jY > s. Therefore theone dimensional POD curve is denoted by: ∀a >0 POD(a) = P(Pro jY > s | a). In this paper oneoffers three different regression models of Pro jY inorder to build an estimation of the POD curve. Nu-merical simulations are computed for the N = 100points of the design of experiments.3.1. Berens method (Berens, 1988)It consists in a linear regression of the outputPro jY . To improve the model, a Box-Cox trans-formation (Box and Cox, 1964) is made on the out-put, which means that we now focus on: yPro jY =Pro jY λ−1λ . λ is determined by maximum likelihood1.07 1.1105001500−0.05EProj Y0.1 0.4050015000.66P11.0 1.6050015000.51iP2Proj Y0.0 0.3050015000.66P20.0 0.6050015000.09ebav1Proj Y0.0 0.605001500−0.08ebav20.0 1.5 3.0050015000.06h1Proj Y0.0 1.5 3.0050015000.08h2Figure 3. Pro jY with respect to the IP.312th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 2015as the real number that offers the finest linear re-gression of yPro jY regarding the parameter a (seeFigure 4).0.1 0.2 0.3 0.4 0.510152025a(ProjYλ−1)λFigure 4. Box-Cox transformation with parameterλ = 0.3 for the response Pro jY .The model is now based on yPro jY and is definedasyPro jY (a) = β0 +β1a+ ε, (1)with ε the model error such as ε ∼ N(0,σ 2ε).Maximum likelihood method provides the es-timators ˆβ0, ˆβ1 et σˆε . Hence the model im-plies the following result: ∀a > 0, yPro jY (a) ∼N(ˆβ0 + ˆβ1a, σˆε 2). Then the value of the PODcurve can be estimated as displayed in the Figure5.We finally get the one dimension POD curve (seeFigure 6). By considering the error that is providedby the property of a maximum likelihood estima-tor in a case of a linear regression, we can use thisuncertainty on both β0 and β1 to build confidenceintervals. The 95% confidence curve that we haveon the estimated POD curve is also illustrated inFigure 6.3.2. Binomial-Berens mix methodHere we keep the linear regression on yPro jY , whichis: ∀a > 0 yPro jY = ˆβ0 + ˆβ1a+ ε but we do notassume that ε is Gaussian anymore. However theerrors are still assumed to be independent and iden-tically distributed. We then consider that we have N0.1 0.2 0.3 0.4 0.510152025a(ProjYλ−1)λSFigure 5. Linear model illustration. The Gaussianpredictive distributions for a = max(P1,P2) = 0.2,0.3 and 0.4 are given. The horizontal linerepresents the detection threshold.0.1 0.2 0.3 0.4 0.5 0.60.00.20.40.60.81.0aPOD(a)Figure 6. Example of POD curve estimation andconfident interval with Berens method.of its realizations which we regroup in the follow-ing vectorεN = yNpro jY − ˆβ0 − ˆβ1aN. (2)Therefore we build its histogram and we add it tothe prediction of the linear model as shown in Fig-ure 7. By using the i.i.d. property of ε , let usconsider that we have N realizations of the randomvalue yPro jY (a) for a > 0 and we can use them toestimate the probability for yPro jY (a) to exceed thethreshold s (see Figure 7).412th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 20150.1 0.2 0.3 0.4 0.510152025a(ProjYλ−1)λSFigure 7. Binomial-Berens method: Berensmethod without normal hypothesis. The Gaussiandensities are replaced by the sample histogram.For each a > 0, let Ns(a) be the number of re-alizations of the random variable yPro jY (a) that arehigher than s. That is to say:Ns(a)=Card({(εi)i∈{1,...,N} | ˆβ0 + ˆβ1a+ εi > s}).(3)Therefore an estimation of POD(a) is given byNs(a)N , with Ns(a) ∼ B (N,POD(a)). The assump-tion on Ns(a) distribution can then be used to buildconfidence intervals on the value of POD(a), fora > 0.3.3. Kriging methodAs some criticism could be made at some point re-garding the i.i.d. property of the model error ε , letus set a Gaussian process regression (Sacks et al.,1989; Fang et al., 2006) in order to build a surrogatemodel of the transformed output yPro jY . Now theinfluence of the other inputs (described in section2) are explicitly mentioned in the model whereas itused to be all included in ε . That is why we nowconsider the set of the most influent inputsx = (E a ebav1 ebav2 h1 h2) . (4)Since the linear trend used in the two previousmethods was relevant, we keep it as the mean of theGaussian process that we are about to use. There-fore, the model is defined as follows:yPro jY (x) = β0 +β1a+Z(x), (5)where Z is the centered Gaussian process. We makethe assumption that Z is second order stationarywith variance σ 2. Besides, we assume that k(·, ·)is the Matérn 5/2 kernel, which is parameterizedby its lengthscale θ (∈ R6 in this case). Thanksto the maximum likelihood method, we can esti-mate the values of the so far-unknown parameters:β0, β1, σ 2 and θ .Kriging provides an estimator of yPro jY (x) whichwe write ŷPro jY (x). Moreover, through its varianceσ 2Z(x), kriging quantifies the uncertainty inducedby estimating yPro jY (x) with ŷPro jY (x). Indeed, onehas the new probability distribution:∀x(yPro jY (x) | yNPro jY)∼ N(ŷPro jY (x),σ 2Z(x))(6)where ŷPro jY (x) is the kriging mean (i.e.E[yPro jY (x) | yNPro jY]) and σ 2Z(x) the krigingvariance. They can both be explicitly estimated.Hence we can estimate the value of POD(a), fora > 0:POD(a)' P(N(ŷPro jY (x),σ 2Z(x))> s | a). (7)By using the uncertainty implied by the Gaussiandistribution regressions, one can build new confi-dence intervals as it is illustrated in Figure 8. Wevisualize the confidence interval induced by theMonte carlo estimation, the one induced by thekriging approximation and the total confidence in-terval (including both approximations).4. CONCLUSIONSThis paper has presented different techniques toProbability of Detection (POD) curves determina-tion (flaw detection probability) in a context of NonDestructive Testing (NDT) procedures. As part ofthis study, we focus on the examination under wearanti-vibration bars of steam generator tubes withsimulations performed by the finite-element com-puter code Code_Carmel3D. The model parameter-ization and the design of numerical experiments areexplained.For the POD curves determination, the Berensmethod, based on a linear regression model, isfirstly studied. It has to be noted that themethod to get confidence intervals on the POD512th International Conference on Applications of Statistics and Probability in Civil Engineering, ICASP12Vancouver, Canada, July 12-15, 20150.1 0.2 0.3 0.4 0.50.00.20.40.60.81.0aPOD(a)PODMC 95% CIPG 95% CIPG + MC 95% CIFigure 8. Example of POD curve estimated with akriging model.first introduced in MIL-HDBK-1823A (2009) andGandosi and Annis (2010) was identified and cor-rected by the authors. In addition, the normalityassumption on residuals required for this methodcan be impaired. Second, we propose an alter-native strategy to both build POD curves and as-sess confidence intervals without assuming Gaus-sian residuals. We compare it with the standardBerens method. In both cases mentioned above, thePOD construction methods are based on a linear-ity assumption. We then present a third approachbased on the non-linear model of the Gaussian pro-cess regression.It is important to remember that all the PODcurves that we got are only examples as long as theinputs are to be validated. Nevertheless these threemethods were proved to be valuable tools to evalu-ate POD curves over a wide range of problems. Fur-ther works will be performed on developing sensi-tivity analysis methods (Iooss and Lemaître, 2015)devoted to POD curves.5. REFERENCESBerens, A. (1988). NDE reliability data analysis,Vol. 17. Metals Handbook, 9th edition, 689–701.Box, G. and Cox, D. (1964). “An analysis of transfor-mations.” Journal of the Royal Statistical Society, 26,211–252.E. de Rocquigny, N. Devictor, and S. Tarantola, eds.(2008). Uncertainty in industrial practice. Wiley.Fang, K.-T., Li, R., and Sudjianto, A. (2006). Designand modeling for computer experiments. Chapman &Hall/CRC.Gandosi, L. and Annis, C. (2010). “Probability of de-tection curves: Statistical best-practice.” ENIQ TGRTechnial Document, 41.Iooss, B. and Lemaître, P. (2015). “A review on globalsensitivity analysis methods.” Uncertainty manage-ment in Simulation-Optimization of Complex Sys-tems: Algorithms and Applications, C. Meloni and G.Dellino, eds., Springer.Maurice, L., Costan, V., Guillot, E., and Thomas, P.(2012). “Eddy current NDE performance demon-strations using simulation tools.” Review of Progressin Quantitative Non Destructive Evaluation, Denver,Colorado, USA, 32, 464–471.MIL-HDBK-1823A (2009). “Nondestructive evaluationsystem reliability assessment.” Department of De-fense Handbook http://mh1823.com/mh1823.Morokoff, W. and Caflisch, R. (1995). “Quasi-MonteCarlo integration.” Journal of Computational Physics,122, 218–230.Pasanisi, A. and Dutfoy, A. (2012). “An industrialviewpoint on uncertainty quantification in simula-tion: Stakes, methods, tools, examples.” Uncertaintyquantification in scientific computing - 10th IFIPWG 2.5 working conference, WoCoUQ 2011, Boul-der, CO, USA, August 1-4, 2011, A. Dienstfrey andR. Boisvert, eds., Vol. 377 of IFIP Advances in In-formation and Communication Technology, Berlin:Springer, 27–45.Sacks, J., Welch, W., Mitchell, T., and Wynn, H. (1989).“Design and analysis of computer experiments.” Sta-tistical Science, 4, 409–435.6
- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015) /
- Building probability of detection curves via metamodels
Open Collections
International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP) (12th : 2015)
Building probability of detection curves via metamodels Browne, Thomas; Le Gratiet, Loïc; Blatman, Géraud; Cordeiro, Sara; Goursaud, Benjamin; Iooss, Bertrand; Maurice, Léa 2015-07
pdf
Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.
Page Metadata
Item Metadata
Title | Building probability of detection curves via metamodels |
Creator |
Browne, Thomas Le Gratiet, Loïc Blatman, Géraud Cordeiro, Sara Goursaud, Benjamin Iooss, Bertrand Maurice, Léa |
Contributor |
International Conference on Applications of Statistics and Probability (12th : 2015 : Vancouver, B.C.) |
Date Issued | 2015-07 |
Description | Probability of Detection (POD) curves is a standard tool in several industries to evaluate the performance of Non Destructive Testing (NDT) procedures. However, the classical methods for POD determination rely on strong statistical assumptions (linearity, residuals normality and homoscedasticity). In the context of numerical POD estimation (with data coming from numerical simulations of the system), we study classic and novel model-based approaches. Applications are performed on Eddy Current Non Destructive Examination numerical data. |
Genre |
Conference Paper |
Type |
Text |
Language | eng |
Notes | This collection contains the proceedings of ICASP12, the 12th International Conference on Applications of Statistics and Probability in Civil Engineering held in Vancouver, Canada on July 12-15, 2015. Abstracts were peer-reviewed and authors of accepted abstracts were invited to submit full papers. Also full papers were peer reviewed. The editor for this collection is Professor Terje Haukaas, Department of Civil Engineering, UBC Vancouver. |
Date Available | 2015-05-25 |
Provider | Vancouver : University of British Columbia Library |
Rights | Attribution-NonCommercial-NoDerivs 2.5 Canada |
DOI | 10.14288/1.0076167 |
URI | http://hdl.handle.net/2429/53430 |
Affiliation |
Non UBC |
Citation | Haukaas, T. (Ed.) (2015). Proceedings of the 12th International Conference on Applications of Statistics and Probability in Civil Engineering (ICASP12), Vancouver, Canada, July 12-15. |
Peer Review Status | Unreviewed |
Scholarly Level | Other |
Rights URI | http://creativecommons.org/licenses/by-nc-nd/2.5/ca/ |
AggregatedSourceRepository | DSpace |
Download
- Media
- 53032-Paper_573_Browne.pdf [ 211.97kB ]
- Metadata
- JSON: 53032-1.0076167.json
- JSON-LD: 53032-1.0076167-ld.json
- RDF/XML (Pretty): 53032-1.0076167-rdf.xml
- RDF/JSON: 53032-1.0076167-rdf.json
- Turtle: 53032-1.0076167-turtle.txt
- N-Triples: 53032-1.0076167-rdf-ntriples.txt
- Original Record: 53032-1.0076167-source.json
- Full Text
- 53032-1.0076167-fulltext.txt
- Citation
- 53032-1.0076167.ris
Full Text
Cite
Citation Scheme:
Usage Statistics
Share
Embed
Customize your widget with the following options, then copy and paste the code below into the HTML
of your page to embed this item in your website.
<div id="ubcOpenCollectionsWidgetDisplay">
<script id="ubcOpenCollectionsWidget"
src="{[{embed.src}]}"
data-item="{[{embed.item}]}"
data-collection="{[{embed.collection}]}"
data-metadata="{[{embed.showMetadata}]}"
data-width="{[{embed.width}]}"
data-media="{[{embed.selectedMedia}]}"
async >
</script>
</div>
Our image viewer uses the IIIF 2.0 standard.
To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.53032.1-0076167/manifest