UBC Faculty Research and Publications

Hydrometeorological Short-Range Ensemble Forecasts in Complex Terrain. Part II: Economic Evaluation. McCollor, Doug; Stull, Roland B. Aug 31, 2008

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52383-Stull_AMS_2008_2007WAF2007064.pdf [ 2.26MB ]
JSON: 52383-1.0041840.json
JSON-LD: 52383-1.0041840-ld.json
RDF/XML (Pretty): 52383-1.0041840-rdf.xml
RDF/JSON: 52383-1.0041840-rdf.json
Turtle: 52383-1.0041840-turtle.txt
N-Triples: 52383-1.0041840-rdf-ntriples.txt
Original Record: 52383-1.0041840-source.json
Full Text

Full Text

AUGUST 2008  MCCOLLOR AND STULL  557  Hydrometeorological Short-Range Ensemble Forecasts in Complex Terrain. Part II: Economic Evaluation DOUG MCCOLLOR The University of British Columbia, and BC Hydro Corporation, Vancouver, British Columbia, Canada  ROLAND STULL The University of British Columbia, Vancouver, British Columbia, Canada (Manuscript received 2 August 2007, in final form 10 December 2007) ABSTRACT Two economic models are employed to perform a value assessment of short-range ensemble forecasts of 24-h precipitation probabilities for hydroelectric reservoir operation. Using a static cost–loss model, the value of the probability information is compared to the values of a deterministic control high-resolution forecast and of an ensemble-average forecast for forecast days 1 and 2. It is found that the probabilistic ensemble forecast provides value to a much wider range of hydroelectric operators than either the deterministic high-resolution forecast or the ensemble-average forecast, although for a small subset of operators the value of the three forecasts is the same. Forecasts for day-1 precipitation provide measurably higher value than forecasts for day-2 precipitation because of the loss of skill in the longer-range forecasts. A decision theory model provides a continuous-variable weighting of a user-specific utility function. The utility function weights are supplied by the ensemble prediction system, and the outcome is compared with weights calculated from a deterministic model, from the ensemble average, and from climatology. It is found that the methods employing the full ensemble and the ensemble average outperform the single deterministic model and climatology for the hydroelectric reservoir scenario studied.  1. Introduction Part I of this two-part paper (McCollor and Stull 2008) documents the meteorological evaluation of a short-range ensemble forecast (SREF) system for hydrometeorological applications in complex terrain. A standardized set of evaluation methods, scores, and diagrams (as described in Hamill et al. 2000) was employed to compare and judge the meteorological skill of the SREF system. However, skill and dispersion measures of the meteorology alone do not give a complete assessment of SREF forecast values (Anderson-Berry et al. 2004). Complete evaluation of the benefit of a forecast system should consider the weather sensitivity and decision-making processes of a particular end user or group of users. Societal and environmental impacts are also  Corresponding author address: Doug McCollor, Dept. of Earth and Ocean Sciences, The University of British Columbia, 6339 Stores Rd., Vancouver, BC V6T 1Z4, Canada. E-mail: doug.mccollor@bchydro.bc.ca DOI: 10.1175/2007WAF2007064.1 © 2008 American Meteorological Society  important components of weather forecast evaluation; however, in most cases economic impacts are easier to directly identify and quantify. In a study by Roulston et al. (2006), laboratory experiments documented the ability of nonspecialist weather forecast users to make sound decisions. That study showed how users increase their expected economic reward while reducing their exposure to risk by incorporating quantitative estimates of weather forecast uncertainty into their decisions. We employ economic skill measures to evaluate userdependant hydrometeorological forecasts. Two economic models adapted for the hydroelectric energy sector are examined in this paper: a cost–loss ratio decision model and a utility-maximizing decision theory model. The economic models are applied in an operations assessment at Jordan River, a hydroelectric facility on the southwest coast of Vancouver Island, British Columbia, Canada. Figure 1 includes a photo of the Jordan River site plus three other reservoir facilities in this region of complex terrain. Forecast evaluation using economic models appli-  558  WEATHER AND FORECASTING  VOLUME 23  FIG. 1. Photos of four BC Hydro dams in southwestern BC, Canada. Clockwise from upper left: Jordan Dam, Daisy Dam, Wahleach Dam, and La Joie Dam. Daisy Dam is seen spilling 700 m3 sϪ1 after very heavy rain in mid-October 2003.  cable to the hydroelectric sector is introduced in section 2. Application of the models to probabilistic forecasts from an SREF system is given in section 3. A case study application is presented in section 4, and conclusions are provided in section 5.  2. Forecast evaluation via economic models a. Background In rapid runoff response pluvial watersheds (the subject of this study), foreknowledge of inflow allows optimal reservoir operation under a set of given operating  constraints. Operating constraints may include, for example, optimizing the depth (hydraulic pressure head) to maximize electrical generation, avoiding spill under high-inflow events, maintaining minimum flow requirements under dry conditions to provide adequate water for fish survival and aquatic habitat, and maintaining stationary reservoir levels under fluctuating inflow conditions to enhance recreational usage. Uncertainty in future rainfall, hence uncertainty in future reservoir inflow, poses significant problems for reservoir operators tasked with meeting stated operating constraints. Minimizing future inflow uncertainty  AUGUST 2008  559  MCCOLLOR AND STULL  through the knowledgeable use of probabilistic forecasts can be analyzed with suitable economic models.  b. Cost–loss model Foreknowledge of high inflow allows a reservoir operator to draft (i.e., lower the water level of) a reservoir ahead of the inflow event. This allows the operator to obtain the full value of the water by routing all reservoir water through the turbines and generators before and during the event. Thus, the operator can capture all of the precipitation-induced runoff without being forced to release excess water (and bypass the revenueproducing generators) downstream. This action is colloquially known as “digging a hole in the reservoir” to capture subsequent inflow. In the managed watersheds analyzed in this study, the dominant source of higher inflow is rainfall events produced by landfalling Pacific frontal systems impinging on the mountainous coastal terrain of southwest British Columbia (see Fig. 1 in Part I for a reference map of the region). There is an economic risk associated with operating a reservoir in the manner outlined above. If no action is taken prior to a high-inflow event, the excess water flowing into the reservoir is released or spilled without generating revenue, hence incurring an economic loss from the value of that water. If the operator does in fact generate power in advance of the high-inflow event, an economic cost is entailed involving hydraulic head loss from lowering the reservoir elevation, which may not be offset if the forecast inflow event does not occur. Economic forecast value can be addressed using the concept of a simple (static) cost–loss ratio decision model (Murphy 1977; Katz and Murphy 1997; Richardson 2000; Zhu et al. 2002; Richardson 2003). Sensible weather forecast examples employing a cost–loss ratio decision model in specific weather forecast evaluations are provided in the literature for road weather forecasts (Thornes and Stephenson 2001), air quality forecasts (Pagowski and Grell 2006), severe weather forecasts (Legg and Mylne 2004), precipitation forecasts from a global ensemble prediction system (Mullen and Buizza 2002), precipitation forecasts over the southwestern United States (Yuan et al. 2005), temperature forecasts for the energy sector (Stensrud and Yussouf 2003), and medium-range flood predictions for watersheds in lowrelief terrain (Roulin 2007). A description of the cost– loss model as applied here to reservoir operations is provided below. A reservoir operator has essentially two alternative courses of action to choose from: either draft the reservoir or not. The choice depends partly on the capacity of the reservoir to hold additional water, and largely on the precipitation forecast, since precipitation is highly  correlated with inflow in these particular watersheds (see Fig. 2 for an example of precipitation–inflow correlation). Each action has an associated cost (from head loss) and leads to an economic benefit (minimize lost revenue due to forced spill) depending on the precipitation (hence inflow) that occurs. The task of the reservoir operator is to choose the appropriate action in each instance that will minimize both the cost and the expected loss over the long term. The cost–loss model is especially useful in this study because it shows how probabilistic forecasts can be used in the decision-making process, and provides a measure of the benefit of probabilistic forecasts over deterministic or ensemble-average forecasts. In the case at hand, the forecast user (the reservoir operator) is sensitive to the specific daily precipitation forecast thresholds analyzed in the companion paper: 5, 10, 25, and 50 mm dayϪ1. The four possible combinations of action and occurrence (with the net cost depending on what happened and what action was taken) are summarized in Table 1. A separate analysis will be performed for each precipitation threshold. In Table 1, a is the number of hits (successful event forecasts), b is the number of false alarms (forecast but not observed), c is the number of misses (observed but not forecast), and d refers to correct rejections (neither forecast nor observed). Hits and false alarms both incur a cost of taking preventative action (C), since the operator drafts the reservoir whenever the event is forecast. Misses are associated with a loss due to the lack of prevention (L). Correct rejections incur no expenses either way. The reservoir operator acts to minimize the average long-term loss by taking appropriate action on each occasion. The economic values (V ) of the forecasts are defined in terms of expenses, E, in an equivalent manner as skill scores for meteorological forecasts (Richardson 2003): Vϭ  Eclimate Ϫ Eforecast , Eclimate Ϫ Eperfect  ͑1͒  where the mean climate expense Eclimate is the least expensive of two options—1) always take protective action, thus incurring a constant cost C but never experiencing a loss L, or 2) never take protective action, which involves no cost but will result in total losses equivalent to oL, where o is the climatological baserate probability of the event occurring. Therefore, Eclimate ϭ min(C, oL). Given perfect knowledge of future events, the reservoir operator would take action only when the event was going to occur and, therefore, would only incur costs at the climatological base rate. The mean expense would then be Eperfect ϭ oC.  560  WEATHER AND FORECASTING  VOLUME 23  FIG. 2. Daily precipitation vs daily inflow for the Jordan River on southwest Vancouver Island. (top) The daily inflow (flow rate in m3 dayϪ1) is determined from reservoir elevation and outflow measurements for the Jordan River reservoir system. (bottom) The daily precipitation (mm dayϪ1) was observed at a precipitation gauge in the middle of the watershed. The period of observations is from April 2003 to March 2005. The Pearson correlation coefficient for these two graphs is 0.88, indicating a high correlation between daily precipitation and daily reservoir inflow. For reservoirs throughout the study region the average Pearson correlation coefficient between daily precipitation and daily inflow is 0.70.  The sample mean expense of the forecast is calculated by multiplying the relative frequencies of each of the four possible outcomes listed in Table 1 by the corresponding expense, resulting in Eforecast ϭ  a b c C ϩ C ϩ L, n n n  ͑2͒  where n ϭ a ϩ b ϩ c ϩ d is the total number of forecast–observation pairs. Thus, the economic value Eq. (1) becomes r c ͑a ϩ b͒ Ϫ n n , min͑r, o͒ Ϫ or  min͑r, o͒ Ϫ Vϭ  ͑3͒  where r ϭ C/L is the specific user’s cost–loss ratio. Summarizing the event categorization described by the binary contingency table elements (a, b, c, and d) into hit rate H ϭ a/(a ϩ c), false alarm rate F ϭ b/(b ϩ d), and (as already suggested) climatological base rate o ϭ (a ϩ c)/n allows Eq. (3) to be rewritten as Vϭ  min͑r, o͒ Ϫ F ͑1 Ϫ o͒r ϩ Ho͑1 Ϫ r͒ Ϫ o . min͑r, o͒ Ϫ or  potential economic value for the cost–loss decision model is uniquely determined from relative operating characteristic (ROC) curves of H versus F (see the appendix in Part I). Additionally, the introduction of the forecast user into the determination of forecast value shows that different users of the same forecast will derive different potential benefits depending on that user’s specific cost–loss ratio. For a particular SREF system and a given precipitation threshold, the value V of the forecasts is solely determined by the user’s cost–loss ratio. By plotting V as a function of cost–loss ratio, r, a water resource manager can determine if the full ensemble forecast system provides more value than does a deterministic forecast or the ensemble mean forecast. TABLE 1. Cost–loss contingency table of precipitation forecasts and observations. The number of hits is given by a, b refers to false alarms, c is the number of misses, and d is the number of correct rejections. Actions resulting from a forecast incur a cost C, while losses L result from each observed event that was not forecast. Correct rejections incur no cost and result in no loss. An event is when the precipitation exceeds a threshold.  ͑4͒  The introduction of hit rate (H ) and false alarm rate (F ) into the evaluation of forecast value shows that the  Forecast Not forecast  Observed  Not observed  a (C) c (L)  b (C) d (0)  AUGUST 2008  561  MCCOLLOR AND STULL  For a reliable ensemble forecast system, the maximum value for a particular user will be obtained by choosing a forecast probability equal to the user’s particular cost–loss ratio, r (Richardson 2000, 2003). Also, the maximum value of V always occurs when r ϭ o. Therefore, forecast users whose C/L ratios happen to equal the climatological base-rate frequency o for a particular event gain the most value from the probability forecasts.  c. Decision theory model The cost–loss model described in section 2b is a binary-decision model. Other techniques that incorporate economic decision making for continuous input–output variables (Katz and Murphy 1997; Smith et al. 2001) such as precipitation and power production can also be applied to reservoir planning processes. In certain situations a reservoir operator may be tasked with maintaining a constant-level reservoir under variable inflow conditions. Optimal use of probabilistic forecasts may be analyzed using a maximumutility decision theory economic model. In decision theory models, optimal decisions in uncertain environments are associated with the maximum of expected rewards. Formal decision theory was originally developed in the first half of the twentieth century by mathematicians aiming at modeling the process by which optimal rational decisions are made (Baldi 2001). In this model, a decision maker in an uncertain environment is faced with a choice of possible actions. Each action has an associated consequence. The consequences are not known with certainty but are expressed as a probability of occurrence. The optimal decision-making strategy is to maximize the expected value (i.e., maximize gain or minimize loss), where expected value refers to a weighted average taken over all possible probabilistic states of the environment. Thus, decision theory naturally leads to probabilistic determination, and an ensemble forecast is a method of defining such weights or probabilities. Economic decision theory is based on the concept of utility, in which users can assign a value or utility U to each potential outcome (Katz and Murphy 1997; Smith et al. 2001). A utility function is a mathematical transformation from money to utility, reflecting the perceived worth of each outcome (instead of the absolute worth) to the decision maker. Let the user’s decision be represented by a continuous variable, x, and the outcome be represented by a continuous variable, y. If the probability distribution function (PDF) of y is p( y), then the expected utility for a given decision, E[U](x), is  E͓U͔͑x͒ ϭ  ͵  U͑x, y͒p͑y͒ dy.  ͑5͒  y  The user can make the choice X that maximizes his or her expected utility: E͓U͔͑X͒ ϭ max͕E͓U͔͑x͖͒. x  ͑6͒  The expected utility hypothesis in economics (Laffont 1989) states that the utility of a user facing uncertainty is calculated by considering the utility in each possible state and constructing a weighted average, the weights being the user’s estimate of the probability of each state. For hydrometeorological analysis, the user must define a utility function that describes the benefit or loss incurred based on a particular reservoir operation decision. The uncertainty in the decision is derived from uncertainty in future rainfall, and the probability of such an uncertain rainfall event is determined from the SREF prediction system. The utility function for this hydrologic model is adapted from another renewable energy source (wind generation) utility function as described by Smith et al. (2001). Short-term rainfall, hence reservoir inflow, is a major factor in determining the capacity to supply electrical energy to the grid. The reservoir operator must be able to forecast generation in advance, subject to maintaining a constant-level reservoir (for the benefit of recreational and other users), since failure to provide energy specified in a contract will lead to purchasing energy at market prices to meet a prespecified contractual delivery. The reservoir manager must decide in advance how much energy their operations will supply the grid on the next day, stipulated by a contract energy production, Kc. This energy is sold at a price, Sc. If actual production, Ka, falls short of the contractual amount, the manager must replace the missing energy at a market price, Sm. If the reservoir manager is assumed to be risk neutral, then the applicable utility function is given by U ϭ Sc ϫ Kc  Ka Ն Kc ,  U ϭ Sc ϫ Ka Ϫ Sm ϫ ͑Kc Ϫ Ka͒  Ka Ͻ Kc .  ͑7͒  The expected utility can be evaluated by assigning a probability pk to each potential energy production amount Ka : E͓U͔͑Ka͒ ϭ  ͚p  k  ϫ U͑Ka, k͒.  ͑8͒  k  3. Results a. Cost–loss model The forecast user (in this case a reservoir manager) is faced with the question of how high the forecast prob-  562  WEATHER AND FORECASTING  VOLUME 23  FIG. 3. Forecast value as a function of C/L for day-1 forecasts. Precipitation thresholds are as indicated in the figures. The thin curves represent forecast values calculated for different probability thresholds pt. The thin curves begin with the lowest probability threshold, pt ϭ 1/11 (dotted line), and continue through increasing values of pt ending with the highest threshold, pt ϭ 11/11 (dashed line). The heavy solid line shows the envelope curve of optimal value (where each user chooses the probability threshold that maximizes the value for his or her specific cost–loss ratio).  ability of a particular event needs to be before the threat is great enough to warrant taking preventative action. The decision maker needs to set a threshold probability, pt, and take preventative action only when the forecast probability exceeds pt. By varying pt over 0/M, 1/M, 2/M, . . . 1, where M is the number of ensembles employed in the SREF, a sequence of hit H versus false alarm F rates can be plotted that traces out the ROC curve for the SREF system (see section 4c in Part I). Different value traces, V, are then plotted as a function of the C/L ratio [see Eq. (4) for each distinct point (F, H ) corresponding to each particular pt on the ROC curve]. The value curves for each precipitation threshold are displayed in Fig. 3 for the day-1 forecasts and in Fig. 4 for the day-2 forecasts. In each of the individual V versus C/L ratio graphs shown in Figs. 3 and 4, the envelope curve (displayed with a heavy solid line) shows the optimum value of the SREF system, obtained when each decision maker em-  ploys the probability threshold pt that maximizes V for their specific C/L ratio. As seen in Figs. 3 and 4, the envelope curve is never less than any of the individual pt curves, indicating that no particular single forecast will benefit different users utilizing different C/L ratios. The maximum value (Vmax) of the envelope curve for V occurs at that value of C/L equal to the climatological base rate o, and is seen to shift toward smaller C/L ratios (lower o, hence rarer events) with increasing precipitation threshold. The shape of the Vmax envelope curves for the day-1 forecasts are similar to those for the day-2 forecasts, for all four precipitation thresholds. This indicates little change in relative value among different users employing different C/L ratios in their decision making. However, the Vmax envelope curve is shifted downward slightly in the day-2 forecasts compared to the day-1 forecasts, indicating that, as expected, the measured value of the forecasts decreases as the forecast range is  AUGUST 2008  MCCOLLOR AND STULL  563  FIG. 4. Forecast value as a function of C/L for day-2 forecasts. The information presented is in the same form as in Fig. 3.  extended. If the probabilistic forecasts were extended into the midrange (3–10-day forecasts), the Vmax envelope curve would necessarily collapse as forecast skill approaches climatology [Eq. (1)]. A similar assessment for midrange forecasts (a potential avenue for future research) would indicate how far into the future these forecasts maintain their positive economic value. Plotting the envelope Vmax versus C/L allows a clearer view of how different forecast users benefit from the probabilistic nature of SREF systems, especially when plots of deterministic forecasts and ensemble-average forecasts are included for comparison. In Figs. 5 and 6 (for day-1 and day-2 forecasts, respectively), the maximum potential value curves are plotted along with a best deterministic forecast and an ensemble-average forecast. The best deterministic forecast was chosen among the individual SREF members as the one with the best error characteristics (see section 4a in Part I), namely the very high-resolution MM5–4 km, model. The ensemble-average forecast is the equally weighted average of all SREF system mem-  ber forecasts. The ensemble-average forecast is included for comparison, although in practice caution is advised in using this forecast because it may not represent a physically realizable solution to the model equations (in the manner that the average of the numbers on a six-sided die, 3.5, is not a physically realizable outcome of rolling the die). The plots in Figs. 5 and 6 indicate that the SREF system provides the best economic value for both day-1 and day-2 forecasts, all precipitation thresholds, and all C/L ratios. Also, these figures show positive economic value to all users at the highest precipitation threshold of 50 mm dayϪ1. However, for lower precipitation thresholds, reservoir managers employing the very lowest C/L ratios progressively lose value as the precipitation threshold lowers from 25 to 10 to 5 mm dayϪ1. The advantage of a probabilistic forecast system is evident from the fact that the EPS exhibits a larger range of forecast users that can economically benefit from the forecasts than either the deterministic forecasts or the ensemble-average forecasts (namely, the dotted and  564  WEATHER AND FORECASTING  VOLUME 23  FIG. 5. Forecast value envelope as a function of C/L for day-1 forecasts. Precipitation thresholds are as indicated in the figures. The dashed line is the deterministic high-resolution forecast and the dotted line is the ensemble-average forecast (the abscissa is plotted with a logarithmic axis for C/L to enhance clarity). Since the maximum value occurs at the climate frequency of the threshold event, the curves for the more extreme events are concentrated around lower C/L values.  dashed curves are narrower and lower than the ensemble envelope curve). For the higher precipitation thresholds of 25 and 50 mm dayϪ1, the ensemble-average forecast outperforms the deterministic forecast in terms of value; however, the probabilistic forecasts are even more advantageous as they extend the valuable range of the forecasts to a broader cast of users. At the lowest precipitation threshold of 5 mm dayϪ1, the measured maximum values Vmax of the SREF system, the deterministic forecast, and the ensemble-average forecast are the same. As the thresholds and their impacts on the reservoir operations increase, the measured maximum value Vmax of the SREF system becomes greater than that of the deterministic and ensemble-average forecasts. Information about choosing the correct probability threshold pt for each decision maker can be shown by plotting the value V as a function of pt for different C/L ratios (see Fig. 7 for the day-1 forecasts and Fig. 8 for the day-2 forecasts). The plots show that reservoir managers with low C/L ϭ 0.2 who have low costs but po-  tentially high losses benefit from taking action (pregenerating electricity and thus lowering the reservoir in advance of a forecast precipitation event) at a precipitation threshold exceedance probability as low as 10%. As the C/L ratio increases to 0.6 for specific reservoirs, the managers should wait until the probability of an event is 30% before taking preventative action. For managers of a reservoir with C/L ϭ 0.8, the associated high cost of lowering the reservoir in advance of a forecast inflow event and limited potential losses from the forced release or spill mean that they should wait until forecasts are in the 70%–80% range before pregenerating. Day-1 SREF system forecasts are measurably better than day-2 forecasts. The plots in Figs. 7 and 8 indicate the trend that the probability of an event in the day-2 forecast time frame should be about 10% higher than the probability in the equivalent day-1 forecast to ensure that the same value ensues from action taken. In other words, reservoir managers operating a reservoir with C/L ϭ 0.2 could pregenerate in the day-2 time frame when the precipitation event forecast probability  AUGUST 2008  MCCOLLOR AND STULL  565  FIG. 6. Forecast value envelope as a function of C/L for day-2 forecasts. The information presented is in the same form as in Fig. 5.  reaches 20%. Managers operating a reservoir with C/L ϭ 0.8 can take preventative action 2 days before a forecast event of 5 or 10 mm dayϪ1, but the probability of occurrence must be close to 90%. This effect is less evident for 25 mm dayϪ1 precipitation events and is nonevident for 50 mm dayϪ1 events, likely because the smaller number of cases for these climatologically rarer events introduces sampling errors into the analysis (as shown in section 4b in Part I). Looking more closely at the relation between precipitation threshold and value for different SREF combinations, Figs. 9 and 10 show this relation for the day-1 forecasts and day-2 forecasts, respectively. The fullensemble SREF provides value for a greater range of users at all precipitation thresholds, compared to the deterministic forecast and the ensemble-average forecast. As the precipitation threshold increases, the forecasts provide value to more and more users toward lower C/L values. There are specific links between the economic value analysis of the cost–loss model and the ROC analysis presented in section 4c in Part I. Figure 11 shows the ROC curve for the day-1 forecasts with a 10 mm dayϪ1  threshold alongside the corresponding maximum value envelope curve. The deterministic high-resolution forecast and ensemble-average forecast are each represented by single points in the ROC diagram. These points are shown connected with straight-line segments to the upper-right and lower-left corners of the ROC diagram. The deterministic forecast point and the ensembleaverage forecast point both lie near the probability forecast ROC curve. This suggests that basic forecast system performance is similar for all three forecast systems. The benefit of the probability forecast depends on the varying needs of different users. As seen on the value curve, all three forecast systems show similar value for a narrow range of users with C/L ratios between about 0.3 and 0.4. For C/L values outside this range, the ensemble forecasts prove most economically valuable. An example using ensemble precipitation forecasts in a cost–loss model for specific reservoirs is given in section 4a(1).  b. Decision theory model The utility functions calculated from Eq. (7) for Jordan River operations indicate the variation of utility  566  WEATHER AND FORECASTING  VOLUME 23  FIG. 7. Forecast value as a function of probability threshold for day-1 forecasts. Precipitation thresholds are as indicated in the figures. Different cost–loss ratios are shown as a solid line (C/L ϭ 0.2), dashed line (C/L ϭ 0.4), dotted line (C/L ϭ 0.6), and dashed–dotted line (C/L ϭ 0.8). A value of 1.0 indicates perfect forecasts for users while values below zero indicate that climatological forecasts are more valuable. The graphs show that users with higher C/L ratios must wait for a higher probability forecast of the given event from the suite of ensembles (i.e., the likelihood of the event occurring is more certain) before taking action to receive value from the forecasts.  with pricing structure. The particular utility function explored here displays increasing penalties as the Sm /Sc ratio increases, and increasing benefits as higher energy contract amounts are selected. The penalties and benefits increase jointly with the precipitation threshold indicated. An example using ensemble precipitation forecasts to maximize the expected value function for Jordan River is given in section 4b.  a simplification of reservoir management and operational constraints, and is intended to examine the value of probabilistic precipitation forecasts for inflow into reservoirs. The power produced from the generator of a hydropower facility depends on the net hydraulic head available to the turbine and the flow through the hydraulic conveyance facility (e.g., canal, intake, or penstocks; see Fig. 12), as given by the following equation (Gulliver 1991):  4. Case studies a. Cost–loss model 1) MODEL  DEVELOPMENT  In this section we provide a practical example of the cost–loss model evaluation for specific watersheds in southwest British Columbia. The case study represents  P ϭ ␩␥Qt H,  ͑9͒  where P ϭ generator power output (W), ␩ ϭ turbine/generator efficiency as a fraction, ␥ ϭ specific weight of water ϭ 9807 NmϪ3 at 5°C, Qt ϭ flow through the turbine (m3 sϪ1), and  AUGUST 2008  MCCOLLOR AND STULL  567  FIG. 8. Forecast value as a function of probability threshold for day-2 forecasts. The information presented is in the same form as in Fig. 7.  H ϭ difference in elevation (termed “head”) between the reservoir level and turbine (m). The energy, K (J), produced by the generator operating for time T (s) under constant flow conditions Qt is given by K ϭ PT.  ͑10͒  The market value, ␯ ($) of this energy is the quantity of energy K times the selling price per unit value of energy, S ($ JϪ1):  ␯ ϭ KS ϭ PTS.  ͑11͒  Electric energy is typically bought and sold in units of either kilowatt hours (kWh) or megawatt hours (MWh), and the price of electrical energy is typically quoted in dollars per MWh. The conversion factor between joules and MWh is 1 MWh ϭ 3.6 ϫ 109 J. We will use the following general variables to describe the hydrometeorological forecast inflow model (we use a time increment of 1 day for all measurements, calculations, and forecasts):  h2 ϭ the difference between the lowered reservoir elevation and the low-level intake (m), h3 ϭ the difference between the reservoir at full operating level and the lowered reservoir elevation prior to an expected inflow event (m), and Qs ϭ the water that spills past the generator (m3) without producing any power (hence any revenue). We will also use the following reservoir-specific constants in describing the model: h1 ϭ nominal head [the height difference between the low-level intake and the turbine (m)], Qb ϭ the daily base inflow (m3), Ar ϭ the surface area of the reservoir (m2), and G ϭ the space in the reservoir available for storage (m3) ϭ (h2 ϩ h3) ϫ Ar. When the reservoir is full, H ϭ h1 ϩ h2 ϩ h3. If the reservoir is lowered an amount h3 in anticipation of a rainfall-induced inflow event, the overall head H will be  568  WEATHER AND FORECASTING  VOLUME 23  FIG. 9. Each bar in the figure spans the portion of the C/L range having positive value for different SREF configurations and for different precipitation thresholds for the day-1 forecast. The bars are labeled with the SREF configurations for the full ensemble, the very-high resolution deterministic forecast, and the ensemble-average forecast. Longer bars provide value to a wider range of users.  reduced to H ϭ h1 ϩ h2, resulting in less power production. The cost–loss ratio for a particular reservoir is determined in the following way. The loss that would ensue if the reservoir was full during an inflow event is the value of the water Qs that spills past the generator without producing any power (hence any revenue). The amount of loss, L ($), would be L ϭ PLTS ϭ ␩␥Qs͑h1 ϩ h2 ϩ h3͒TS,  ͑12͒  where PL is lost power and Qs is the amount of water spilled if the reservoir is not lowered a distance h3: Qs ϭ Ar ϫ h3.  ͑13͒  The cost of preventing the reservoir from spilling results from operating the reservoir at a lower head, h1 ϩ h2, in anticipation of an inflow event, versus power produced at h1 ϩ h2 ϩ h3 (effectively operating at a net head of h3): C ϭ ␩␥Qt h3TS.  ͑14͒  The flow through the turbine, Qt , is at least equal to the baseflow Qb. The C/L ratio is then  CրL ϭ  Qb Qbh3 ϭ . Ar h3͑h1 ϩ h2 ϩ h3͒ Ar h1 ϩ G  ͑15͒  The C/L ratio can then be calculated for individual reservoirs. See Table 2 for the physical parameters of the reservoirs included in the study, and Table 3 for the associated ratios for this basic C/L model. Equation (15) shows that small C/L ratios are associated with reservoirs with large surface areas, high head, large storage, and low baseflow. This type of reservoir is typically found in regions of high mountains and deep valleys where most of the reservoir inflow stems from precipitation-generated surface runoff (climatically high-rainfall zones). Conversely, large C/L ratios are associated with reservoirs with small surface areas, low head, low storage, and high baseflow. This type of reservoir would typically be found in areas of low, rolling relief where much of the reservoir inflow derives from subsurface aquifers (climatically lowrainfall zones). The reservoirs included in this study are of the former subtype and are characterized by very low C/L ratios. We can further refine the C/L ratios calculated by Eq. (15). The cost of operating a reservoir at a lower  AUGUST 2008  569  MCCOLLOR AND STULL  FIG. 10. Each bar in the figure spans the portion of the C/L range having positive value for different SREF configurations and for different precipitation thresholds for the day-2 forecast. The information presented is in the same form as in Fig. 9.  head may be realized for multiple days, until the next precipitation event occurs. The length of time until the next precipitation event is estimated by 1/ o, the inverse of the climatological frequency of the event. In this manner, the C/L ratio is defined as  CրL ϭ  Qb . o͑Ar h1 ϩ G͒  ͑16͒  A further refinement in the cost–loss model is realized by eliminating the constraint of a constant price of  FIG. 11. The relationship between ROC and forecast value. Curves are for day-1 forecasts with precipitation thresholds of 10 mm dayϪ1. (left) The ROC curve with the SREF probability threshold pt square markers corresponds to (right) the thick solid envelope value curve. The dashed lines in both figures are the very-high resolution deterministic forecasts and the dotted lines in both figures are the ensemble-average forecasts. The open circles on the dashed line and the dotted line in the figure on the left represent the very high-resolution deterministic forecast point and the ensemble-average forecast point, respectively. The relative merit of ROC and forecast value as determined by these curves is discussed in the text.  570  WEATHER AND FORECASTING  VOLUME 23  FIG. 12. The reservoir schematic diagram for the cost–loss economic model discussed in the text.  energy, S. In real-time energy markets, the price of energy fluctuates hourly so that energy lost at one time and recouped at another time may not have the same value. In the cost–loss model examined here, the energy  lost through spill may be valued at a contract price, Sc, while the energy lost through head loss must be replaced at a market price, Sm. In this case, the C/L ratio would be  TABLE 2. Reservoir physical parameters for cost–loss calculations.  Index  Reservoir  Daily base inflow Qb (ϫ106 m3)  Nominal head h1 (m)  Reservoir storage G (ϫ106 m3)  Reservoir area Ar (km2)  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  Alouette Buntzen Carpenter Clowhom Comox Coquitlam Daisy Downton Elsie Hayward Lake Jordan River Seton Stave Upper Campbell Wahleach  1.8 2.0 4.4 3.1 2.9 2.0 4.2 3.5 1.8 0.86 1.0 1.6 9.6 6.7 0.60  38 116 340 44 104 116 291 54 224 38 265 45 38 43 573  155 202 928 45 106 202 46 722 77 24 28 9.0 36 82 66  16 1.8 48 8.0 30 12 43 24 0.8 3.0 1.7 25 62 67 4.9  AUGUST 2008  571  MCCOLLOR AND STULL TABLE 3. Cost–loss ratios for the reservoirs in this study.  Index  Reservoir  Basic model  Including o (50 mm dayϪ1 threshold)  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  Alouette Buntzen Carpenter Clowhom Comox Coquitlam Daisy Downton Elsie Hayward Lake Jordan River Seton Stave Upper Campbell Wahleach  0.0024 0.0048 0.000 26 0.0078 0.000 91 0.0012 0.000 34 0.0017 0.0074 0.0063 0.0022 0.0014 0.0035 0.0018 0.000 21  0.074 0.15 0.0080 0.24 0.029 0.038 0.011 0.054 0.23 0.20 0.068 0.044 0.11 0.057 0.0066  CրL ϭ  Sm Qb ϫ . Sc o͑Ar h1 ϩ G͒  ͑17͒  Table 3 also lists the C/L ratios for individual reservoirs (for precipitation thresholds of 50 mm dayϪ1) incorporating the climatological frequency of the event [Eq. (16)] and the variable price of energy [Eq. (17)].  2) EXAMPLE We can provide an estimate of the actual sample expense of the ensemble forecasts by evaluating the elements in Table 1 for a particular set of forecasts for a particular reservoir. In this example we will use the set of two cool season forecasts for Jordan River. The actual sample expense of the forecasts can be calculated in a similar manner as Eq. (2): Eactual ϭ a ϫ C ϩ b ϫ C ϩ c ϫ L ϩ d ϫ 0,  ͑18͒  TABLE 4. Cost–loss contingency table summary of actual precipitation forecasts and observations for Jordan River for the 50 mm dayϪ1 precipitation threshold. The summary includes the full ensemble, the deterministic forecasts from the MM5–4 km model, and the ensemble average. The actual sample expense, Eactual, is determined as a function of L through the cost–loss ratio for Jordan River (C/L ϭ 0.068). The cost is estimated by evaluating L as explained in the text.  Full ensemble Deterministic Ensemble average  Hit (a)  False alarm (b)  Miss (c)  Eactual  Cost estimate ($)  4 4 2  0 4 0  3 3 5  3.27L 3.54L 5.14L  955,000 1,035,000 1,500,000  Including market price value Sm /Sc ϭ 2.5 (50 mm dayϪ1 threshold) 0.18 0.37 0.020 0.61 0.071 0.094 0.026 0.13 0.58 0.50 0.17 0.11 0.27 0.14 0.016  where a, b, c, and d are the elements in Table 1 (also called hits, false alarms, misses, and correct rejections, respectively). The summary of the cost–loss table elements for Jordan River and a 50 mm dayϪ1 precipitation threshold are given in Table 4. Separate calculations are included for the full ensemble, the ensemble average, and the best deterministic forecast. The cost– loss ratio for Jordan River that incorporates the climatological frequency of the 50 mm dayϪ1 event and a constant market price for energy is C/L ϭ 0.068. This C/L ratio is used for the calculations. The turbine/ generator efficiency, ␩, is assumed to be 0.90 (Gulliver 1991), and the cost of energy is assumed constant at $50 per MWh.1 Dollar value estimates for L can be calculated from Eqs. (12) and (13). The resulting cost estimates for C/L operating decisions at Jordan River based on the 50 mm dayϪ1 precipitation threshold are given in Table 4. The full ensemble provides a net benefit (lower cost) of $80,000 ($40,000 annually) or 8% improvement compared to the deterministic forecasts, and a net benefit of $545,000 ($272,500 annually) or 57% improvement compared to the ensemble average forecasts.  b. Decision theory model In this section we provide a practical example of the decision theory model for a specific watershed in southwest British Columbia. As with the cost–loss model, this case study represents a significant simplification of 1  Source: U.S. Energy Information Administration (2005), Form EIA-861, Annual Electric Power Industry Report. The price quoted is the latest available average wholesale price of electricity for the United States.  572  WEATHER AND FORECASTING  VOLUME 23  FIG. 13. Profit as a function of the market–contract price ratio for different forecast configurations. The profit for each different configuration is calculated relative to a baseline maximum possible profit (if the weather forecasts were perfect).  reservoir operating decisions and is intended to examine the value of probabilistic precipitation forecasts derived from SREF-generated ensembles. The energy produced by the reservoir outflow, Qout, is given by Eq. (10). If the constraint to maintain a level reservoir is met, then Qout ϭ Qin. The day-1 forecasts of 24-h precipitation and associated daily inflow observations from Jordan River on southwest Vancouver Island were used in this analysis. The linear Pearson correlation coefficient for Jordan River is 0.88 (see Fig. 2). Therefore, the hydrologic response (inflow) to rainfall can be approximated by a linear function: Qin ϭ ␬R ϭ Qout ϭ Qt. The energy, K, is evaluated from Eqs. (9) and (10): K ϭ ␩␥␬RHT.  ͑19͒  The utility function [Eq. (7)] can then be evaluated in terms of the energy produced from different precipitation events, R. The probabilities assigned to forecasting different precipitation amounts, pk, are derived from the SREF system forecasts of 24-h precipitation for Jordan River, and the resultant expected value is calculated from Eq. (8). The results (using day-1 forecasts)  are shown in Fig. 13 where profit2 (relative to perfect forecasts) is shown as a function of the market price to contract price ratio, Sm /Sc. For comparison purposes, Fig. 13 also includes the profit from operating the reservoir using climatology to assign probabilities, as well as the SREF ensemble average and the deterministic forecasts from the MM5–4 km forecast model. The results show that when the contract price is equivalent to the market price, there is little difference whether using the full ensemble, the ensemble average, or the deterministic single forecast to provide probability estimates. Note that using climatologic representative probabilities results in a loss for all values of Sm /Sc. However, as the spread between the market price and contract price widens and the Sm /Sc ratio increases, a different profit scenario emerges. The ensemble average forecast provides the greatest profit, followed closely by the full 11-member ensemble. The profit re-  2 Profit as used in this context is the difference between generation revenue and contract costs for undelivered energy. There are many other costs associated with hydropower generation, such as water license fees and transmission charges, that are not included in the profit calculations here.  AUGUST 2008  573  MCCOLLOR AND STULL  alized by using the deterministic high-resolution forecast drops quickly relative to the SREF forecasts.  5. Conclusions Weather forecasts are made and used under the assumption that forecast users can take preventative action, prior to a predicted weather event, that provides more benefit to the user than having no forecast at all. This assumption should be specifically tested to aid weather forecast providers in improving their forecasts, and to assist weather forecast users in assessing the benefit of the forecasts to them. Forecasts can be tested by assessing both their meteorological skill and their economic value. Forecast skill is generally determined by assessing the accuracy of a forecast-producing system relative to a given standard such as climatology, persistence, or another forecasting system. Forecast skill will vary among different forecast-producing systems, but remains user independent. Part I assessed the meteorological skill of a shortrange ensemble forecast system designed to provide 1and 2-day precipitation forecasts in complex mountainous and coastal terrain. Forecast value is a separate assessment of a forecasting system that takes into account diverse needs and benefits attributed to different forecast users. This paper comprises an economic value assessment of the same set of forecasts that provided the basis for the meteorological skill assessment described in Part I. The particular forecast users studied here are water resource managers tasked with making efficient and effective use of rainfall that ultimately drives hydroelectric power in rainfall-fed mountain–coastal reservoirs. Employing a static cost–loss economic model adapted to reservoir operations for hydroelectric managers, a forecast system that provides a suite of ensembles yields positive economic value for a broader range of users than do either a single fine-resolution forecast model or a single forecast composed of the ensemble average. Second, the study showed that different forecast users, characterized by specific cost–loss ratios, require different precipitation probability thresholds to trigger preventative actions. Users with different cost–loss ratios correspond to operators of reservoirs with different head-loss parameters, different sizes of reservoirs, and different baseflows (e.g., lower C/L ratio applies to reservoirs with low base inflows, high head, large areal reservoirs, and high storage, all of which are typical of rainfall-fed mountain watersheds). The larger economic benefit of probabilistic forecasts is achieved by allowing users to match forecast probabilities with cost–loss ratios specific to their particular operational needs.  Decision theory, employing a utility function designed specifically for market selling of hydroelectric energy from rapid response rainfall-dominated reservoirs, was used to quantify the positive impact (profit generation) from employing an EPS to generate probabilistic forecasts. The forecasts generated by the SREF system analyzed in Part I are superior to a single, highresolution deterministic forecast in maximizing profit from the particular hydroelectric facility (Jordan River) studied here, with the ensemble-averaged forecasts outperforming the full suite of ensembles by a relatively small margin. Employing economic models such as a static binary cost–loss model, or a continuous decision theory model, is a preliminary step in assessing the value of ensemblebased precipitation forecasts to water resource managers. The ultimate goal of ensemble forecasting for reservoir operating planners is to be able to translate the uncertainty in future weather into uncertainty in runoff and inflow, and ultimately the amount of energy generated at a particular reservoir/hydroelectric facility. The uncertainty in overall inflow must also incorporate uncertainty in the hydrologic model that translates forecast precipitation and temperature into reservoir inflow. Future research will include a hydrometeorological model driven by probabilistic meteorological input variables to determine inflows and, ultimately, available energy. Acknowledgments. The authors thank BC Hydro Corporation and the Meteorological Service of Canada for providing the precipitation observations necessary to perform this study. In addition, the Geophysical Disaster Computational Fluid Dynamics Centre (GDCFDC), within the Department of Earth and Ocean Sciences at the University of British Columbia (UBC), is acknowledged for providing the numerical model forecasts used in this study. In particular, we thank Mr. George Hicks II for his efforts in making the numerical forecast information available to us. The GDCFDC gratefully acknowledges the following organizations for computer-purchase grants: Canadian Foundation for Innovation, British Columbia Knowledge Development Fund, and UBC. Additional grant support was provided by the Canadian Natural Sciences and Engineering Research Council, and Environment Canada. The authors thank the two anonymous reviewers for their suggestions to improve the paper. REFERENCES Anderson-Berry, L., T. Keenan, J. Bally, R. Pielke Jr., R. Leigh, and D. King, 2004: The societal, social, and economic impacts of the World Weather Research Programme Sydney 2000  574  WEATHER AND FORECASTING  Forecast Demonstration Project (WWRP S2000 FDP). Wea. Forecasting, 19, 168–178. Baldi, P., 2001: The Shattered Self: The End of Natural Evolution. MIT Press, 259 pp. Gulliver, J. S., 1991: Hydraulic conveyance design. Hydropower Engineering Handbook, J. S. Gulliver and R. E. A. Arndt, Eds., McGraw-Hill, 5.1–5.81. Hamill, T. M., S. L. Mullen, C. Snyder, Z. Toth, and D. P. Baumhefner, 2000: Ensemble forecasting in the short to medium range: Report from a workshop. Bull. Amer. Meteor. Soc., 81, 2653–2664. Katz, R. W., and A. H. Murphy, 1997: Economic Value of Weather and Climate Forecasts. Cambridge University Press, 222 pp. Laffont, J.-J., 1989: The Economics of Uncertainty and Information. MIT Press, 303 pp. Legg, T. P., and K. R. Mylne, 2004: Early warnings of severe weather from ensemble forecast information. Wea. Forecasting, 19, 891–906; Corrigendum, 22, 216–219. McCollor, D., and R. Stull, 2008: Hydrometeorological shortrange ensemble forecasts in complex terrain. Part I: Meteorological evaluation. Wea. Forecasting, 23, 533–556. Mullen, S. L., and R. Buizza, 2002: The impact of horizontal resolution and ensemble size on probabilistic forecasts of precipitation by the ECMWF Ensemble Prediction System. Wea. Forecasting, 17, 173–191. Murphy, A. H., 1977: The value of climatological, categorical, and probabilistic forecasts in the cost–loss ratio situation. Mon. Wea. Rev., 105, 803–816. Pagowski, M., and G. A. Grell, 2006: Ensemble-based ozone forecasts: Skill and economic value. J. Geophys. Res., 111, D23S30, doi:10.1029/2006JD007124. Richardson, D. S., 2000: Skill and relative economic value of the  VOLUME 23  ECMWF Ensemble Prediction System. Quart. J. Roy. Meteor. Soc., 126, 649–667. ——, 2003: Economic value and skill. Forecast Verification: A Practitioner’s Guide in Atmospheric Science, I. Jolliffe and D. B. Stephenson, Eds., Wiley, 164–187. Roulin, E., 2007: Skill and relative economic value of mediumrange hydrological ensemble predictions. Hydrol. Earth Syst. Sci., 11, 725–737. Roulston, M. S., G. E. Bolton, A. N. Kleit, and A. L. SearsCollins, 2006: A laboratory study of the benefits of including uncertainty information in weather forecasts. Wea. Forecasting, 21, 116–122. Smith, L. A., M. S. Roulston, and J. von Hardenberg, 2001: End to end forecasting: Towards evaluating the economic value of the ensemble prediction system. ECMWF Tech. Rep. 336, 29 pp. Stensrud, D. J., and N. Yussouf, 2003: Short-range ensemble predictions of 2-m temperature and dewpoint temperature over New England. Mon. Wea. Rev., 131, 2510–2524. Thornes, J. E., and D. B. Stephenson, 2001: How to judge the quality and value of weather forecast products. Meteor. Appl., 8, 307–314. U.S. Energy Information Administration, 2006: Annual electric power industry report. Form EIA-861. Yuan, H., S. Mullen, X. Gao, S. Sorooshian, J. Du, and H.-M. H. Juang, 2005: Verification of probabilistic quantitative precipitation forecasts over the southwest United States during winter 2002/03 by the RSM ensemble system. Mon. Wea. Rev., 133, 279–294. Zhu, Y., Z. Toth, R. Wobus, D. Richardson, and K. Mylne, 2002: The economic value of ensemble-based weather forecasts. Bull. Amer. Meteor. Soc., 83, 73–83.  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items