Science, Faculty of
Earth, Ocean and Atmospheric Sciences, Department of
DSpace
UBCV
McKinney, Reagan
Master of Science - MSc
University of British Columbia
Accurate weather predictions are crucial to the wellbeing of modern society. The Weather Forecast Research Team (WFRT) runs an ensemble of numerical weather prediction models daily (Short Range Ensemble Forecast; SREF 00Z). The ensemble can be combined into an “average” which is usually the best deterministic forecast. In this study, we experiment with ways of improving its skill for southwestern British Columbia. To do this, four different weighting techniques for ensemble averaging are employed using all 51 members of the SREF. These are constructed using 1 year (October 2021-September 2022) of forecasts and observation data pairs from 90 stations in the region, for temperature, wind speed and precipitation. The weights were then applied to a second year of data (October 2022-September 2023) and compared to the current SREF. To investigate the use of a shorter training set on the weights, as suggested by the literature, a sliding window approach was also used with the weights being calculated with the forecast and observation data pairs for the week and month prior to the forecast of interest. We also experimented with varying the number of ensemble members to explore the sensitivity of the SREF to the number of members. Spatially, there were some special cases where the weighted ensembles performed better or poorer than the current SREF. However, all four weighting schemes did not perform statistically significantly compared to the current equally-weighted SREF (pvalue > 0.05). As for the varying number of ensemble members, it was found that the skill of all of the ensembles varied throughout the year and was never consistently in the small or large range for all tested variables. However, we did note that the skill converged for all members greater than 30. This indicated a lower limit of the necessary ensemble members to match the current SREF skill. Future studies should include more data (i.e. longer datasets; climatology) to get a better picture of the state of the SREF and how its members are interacting and performing.
Investigating the utility of weighting schemes on a 51-member short-range ensemble forecast by Reagan McKinney B.A.Sc. Environmental Engineering, University of Waterloo, 2021 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Atmospheric Science) The University of British Columbia (Vancouver) February 2024 © Reagan McKinney, 2024 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: Investigating the utility of weighting schemes on a 51-member short-range ensemble forecast submitted by Reagan McKinney in partial fulfilment of the requirements for the degree of Master of Science in Atmospheric Science Examining Committee: Roland Stull, Professor, Atmospheric Sciences, UBC Supervisor Rachel White, Assistant Professor, Atmospheric Sciences, UBC Supervisory Committee Member Rosie Howard, Research Director, Atmospheric Sciences, UBC Supervisory Committee Member Doug McCollor, Adjunct Professor, Atmospheric Sciences. UBC Additional Examiner iii Abstract Accurate weather predictions are crucial to the wellbeing of modern society. The Weather Forecast Research Team (WFRT) runs an ensemble of numerical weather prediction models daily (Short Range Ensemble Forecast; SREF 00Z). The ensemble can be combined into an “average” which is usually the best deterministic forecast. In this study, we experiment with ways of improving its skill for southwestern British Columbia. To do this, four different weighting techniques for ensemble averaging are employed using all 51 members of the SREF. These are constructed using 1 year (October 2021-September 2022) of forecasts and observation data pairs from 90 stations in the region, for temperature, wind speed and precipitation. The weights were then applied to a second year of data (October 2022-September 2023) and compared to the current SREF. To investigate the use of a shorter training set on the weights, as suggested by the literature, a sliding window approach was also used with the weights being calculated with the forecast and observation data pairs for the week and month prior to the forecast of interest. We also experimented with varying the number of ensemble members to explore the sensitivity of the SREF to the number of members. Spatially, there were some special cases where the weighted ensembles performed better or poorer than the current SREF. However, all four weighting schemes did not perform statistically significantly compared to the current equally-weighted SREF (pvalue > 0.05). As for the varying number of ensemble members, it was found that the skill of all of the ensembles varied throughout the year and was never consistently in the small or large range for all tested variables. However, we did note that the skill converged for all members greater than 30. This indicated a lower limit of the necessary ensemble members to match the current SREF skill. Future studies should include more data (i.e. longer datasets; climatology) to get a better picture of the state of the SREF and how its members are interacting and performing. iv Lay Summary Accurate weather predictions are crucial to the wellbeing of modern society. The Weather Forecast Research Team runs many weather forecasts that are then averaged to get an overall forecast. Here, we hope to improve this average by applying weights to each forecast rather than equally averaging all of the forecasts. Our technique for weighting the forecasts did not significantly improve the forecasts as hypothesized. We also tested how important all 51 forecasts were to the average forecast, and found that the skill of the average forecast plateaus at 30 members. This is important because reducing the number of forecasts run can reduce the computer resources needed. This study only used two years of data which does not represent climatological patterns in the region nor does it have a large sample of extreme events. Therefore, more information is needed before any forecasts are removed. v Preface Dr. Roland Stull developed the original idea for this project. The methodology of calculating weights with a Logistic Function was proposed by Dr. Roland Stull. The methodology for calculating weights as an inverse of their sum was given by Gneiting et al. (2008). All NWP forecasts and observational used in this thesis are archives of past NWP forecasts from WFRT’s operational SREF and archived observational data from the WFRT emergency weather net. To obtain these forecasts, the author adopted NWP forecast retrieval codes originally written by Dr. Henryk Modzelewski and Eva Gnegy. The author performed all data retrieval and analysis and wrote all codes to produce the weighted ensembles and varying number of members ensemble. Dr. Roland Stull, Dr. Rachel White and Dr. Rosie Howard offered guidance, editing, and refinement of procedures. vi Table of Contents Abstract ........................................................................................................................................................ iii Lay Summary ................................................................................................................................................ iv Preface .......................................................................................................................................................... v Table of Contents ......................................................................................................................................... vi List of Tables .............................................................................................................................................. viii List of Figures ............................................................................................................................................... ix Acknowledgements ..................................................................................................................................... xii 1. Introduction .......................................................................................................................................... 1 2. Background ........................................................................................................................................... 2 2.1 Ensembles ........................................................................................................................................... 2 2.2 Ensemble Size...................................................................................................................................... 2 2.3 Training Period .................................................................................................................................... 3 2.4 Weighted Ensemble Techniques ......................................................................................................... 5 2.4.1 Performance-Based Weighting (PBW) .......................................................................................... 6 2.4.2 Bayesian Model Averaging ........................................................................................................... 8 2.4.3 Multilinear Regression (MLR) ....................................................................................................... 9 2.4.4 Clustering ................................................................................................................................... 10 2.4.5 Likelihood Functions (LF) ............................................................................................................ 11 2.4.6 Monte Carlo................................................................................................................................ 11 2.5 Summary of Literature Review ......................................................................................................... 11 3. Methodology ....................................................................................................................................... 13 3.1 Datasets ............................................................................................................................................ 13 3.1.1 Ensemble Members ................................................................................................................... 13 3.1.1 Observations .............................................................................................................................. 15 3.2 Weighting Technique ........................................................................................................................ 15 3.2.1 Logistic Function ........................................................................................................................ 17 3.2.2 Inverse of the Error Metric (IEM) ............................................................................................... 18 3.2.3 Variable Number of Members (VNE) ......................................................................................... 18 3.2.4 Significance testing .................................................................................................................... 18 4. Results ................................................................................................................................................. 19 4.1 Weighted Ensembles ........................................................................................................................ 19 vii 4.1.1 Weights ...................................................................................................................................... 19 4.1.2 Weighted Ensemble Forecasts ................................................................................................... 23 4.1.3 Forecast Verification and Statistical Significance....................................................................... 28 4.2 Iterated-Mean Ensemble .................................................................................................................. 31 5. Discussion ............................................................................................................................................ 35 5.1 Weighting Schemes ........................................................................................................................... 35 5.2 VNE .................................................................................................................................................... 36 5.3 Weighted Ensemble Limitations ....................................................................................................... 36 5.3 Study Limitations .............................................................................................................................. 37 6. Conclusion ........................................................................................................................................... 38 References .................................................................................................................................................. 39 Appendix A. Station Observation Metadata ............................................................................................... 44 Appendix B. Verification Metric Equations ................................................................................................. 46 viii List of Tables Table 1. List of WFRT SREF Member Attributes. ......................................................................................... 14 Table 2. Threshold Values used for Quality Check on Observation Data. .................................................. 15 Table 3. T-Test Results for comparison of temperature verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. ......................................................... 29 Table 4. T-Test Results for comparison of wind speed verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. ......................................................... 30 Table 5. T-Test Results for comparison of precipitation verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. ......................................................... 31 ix List of Figures Figure 1. Histogram of the number of ensemble members used in the literature included in this review, divided into bins having a width of six. ......................................................................................................... 3 Figure 2. Histogram of the training periods used in the literature included in this review. ........................ 4 Figure 3. Breakdown of the most popular techniques for creating weighted ensembles. This included Bayesian Model Averaging (BMA), Multilinear Regression, Likelihood Functions, Clustering, Monte Carlo Simulations, and Performance Based weighting. The latter is broken down further into weighting based on root mean square error (RMSE), the continuous ranked probability score (CRPS), mean square error (MSE) and mean absolute error (MAE). ........................................................................................................ 5 Figure 4. Map of all the domains included in the WFRT SREF. ................................................................... 14 Figure 5. Map of all the observation stations for this study. ...................................................................... 15 Figure 6. Schematic of seasonal weighting scheme.................................................................................... 16 Figure 7. Schematic of sliding window weighting scheme. ........................................................................ 16 Figure 8. Schematic of the Rank Based Logistic Function (RLF) parameters, rcut and rhalf. ......................... 18 Figure 9. Values-Based Logistic Function (VLF) calculated weights for temperature based on the MAE verification metric. The effect of different k-parameters (40, 100, 200) is shown. Weights are specific to fall. .............................................................................................................................................................. 20 Figure 10. Rank-Based Logistic Function (RLF) calculated weights for temperature. rcut and rhalf values are set to 30 and 16, respectively. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. Narrow lines indicate near zero weights. ............................................................................ 20 Figure 11. Rank-Based Logistic Function (RLF) calculated weights for temperature. rcut and rhalf values are set to 15 and 7, respectively. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. Narrow lines indicate near zero weights. ............................................................................ 21 Figure 12. Inverse Error Metric (IEM) calculated weights for temperature. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. ........................................................................ 22 Figure 13. Inverse Error Metric (IEM) calculated weights for precipitation. Weights were calculated using Pierce Skill Score (PSS), Critical Success Index (CSI) and Gilbert Skill Score (GSS) respectively. All are specific to the season of fall. ....................................................................................................................... 22 Figure 14. Spatial comparison of temperature forecasting performance of the equally weighted short-range ensemble forecast (SREF) and three weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; right); and the Sliding Window-weekly (SWeekly; bottom). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. ............................. 23 Figure 15. Spatial comparison of wind speed forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; right); the Sliding Window-weekly (SWeekly; bottom left); and the Inverse Error Metric (IEM; bottom right). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the x weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. ............................................................................................. 24 Figure 16. Spatial comparison of precipitation forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; top left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; top right); the Sliding Window-weekly (SWeekly; bottom left); and the Inverse Error Metric (IEM; bottom right). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. ........................................................................ 25 Figure 17. Temporal comparison of temperature forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-weekly; SW-monthly); and the Inverse Error Metric (IEM). ........................................................................................... 26 Figure 18. Temporal comparison of wind speed forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-weekly; SW-monthly); and the Inverse Error Metric (IEM). ........................................................................................... 27 Figure 19. Temporal comparison of precipitation forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-Weekly; SW-Monthly); and the Inverse Error Metric (IEM). Here, missing bars for all ensembles indicated a calculation error due to zeros in the dataset; whereas, 1-2 bars missing indicated missing data from those ensembles. ........................................................................................................................................ 28 Figure 20. Verification for temperature of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. ................................................................................................... 29 Figure 21. Verification for wind speed of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. ................................................................................................... 30 Figure 22. Verification for precipitation of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. ................................................................................................... 31 Figure 23. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for temperature. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). ...................................... 32 xi Figure 24. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for wind speed. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). ........................................... 33 Figure 25. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for precipitation. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). ...................................... 34 Figure 26. Time series of each member's performance from October 2021 to November 2023 for temperature. ............................................................................................................................................... 35 xii Acknowledgements To start, I must express my gratitude to Prof. Roland Stull for this opportunity and funding to conduct research under his supervision. He offered continuous encouragement for all my endeavors throughout the degree program and constantly supported me in taking on new opportunities to gain skills in my interest areas. I would also like to thank my committee members, Dr. Rachel White, and Dr. Rosie Howard, for their expertise and insight into my research. I would also like to extend a truck load of gratitude to all the members of the Weather Forecasting Research Team for providing me with constant knowledge, inspiration, and friendship. Major thanks are also extended to Tim Chui, Dr. Henryk Modzeleski and Roland Schigas for their constant IT support and expertise in numerical weather prediction. I would like to acknowledge MITACS and BC Hydro for providing my funding for this research and to the weather forecasting research team for funding all the computer and office resources. Lastly, I would like to thank all my friends and family for their constant encouragement throughout this process and their unwavering belief in me to accomplish this feat. 1 1. Introduction Improving numerical weather prediction is vital for all aspects of life. Weather forecasts provide insight into how, when and if our water resources will be replenished; how, when and if forest fires will pose a serious threat; and how renewable energy sources such as wind, solar and hydro will provide energy to the grid in the short-term, as well as the need for energy during this time due to heatwave or cold spell. In short, the weather affects many aspects of our lives, making predictions of its movements of the utmost importance. This has been a well-recognized notion for centuries; however, only the birth of computers and weather models have been able to make modern operational forecasts a reality. Numerical weather forecasts began as single models. However, forecasters soon realized that due to the highly stochastic nature of the weather, these single models led to too much uncertainty (Lorenz, 1965). Therefore, weather ensemble forecasts were created. A weather ensemble forecast is an agglomeration of several different model configurations or several different initial conditions. Including both different parameterization schemes and initial conditions allows the ensemble to converge on a deterministic forecast while accounting for uncertainty in both these areas. The literature is unanimous in that these ensembles perform much better than individual models in every case (Gneiting and Raftery, 2005). This is because of the ensemble’s ability to account for a portion of the uncertainty. Further, is has been shown in many studies that an increase in the number of members will, on average, always increase the skill of an ensemble. By including a varying number of members in an equally weighted ensemble, we hope to shed more light on this finding. Ensemble averages consisting of equal weighting of all the members includes poor performing members and higher performing members. We hypothesize that the ensemble average can be improved with unequal weighting of the ensemble members. In this study we attempt to answer the following questions: 1. Do ensemble averages with optimized weights outperform the equally weighted ensemble? 2. How sensitive is an ensemble average to the number of members? 2 2. Background 2.1 Ensembles Ensembles were developed to deal with the uncertainty and chaotic nature of the atmosphere in numerical weather prediction. In 1965, Lorenz noted that numerical weather prediction is near impossible due to the high uncertainty in the initial conditions and the high sensitivity of the atmospheric response to the initial conditions (chaos). As a result, numerical weather prediction evolved to use ensemble members that include several different perturbations of initial conditions, physics parameterizations, dynamical cores, numerics, and grid sizes. The term “member” is used to describe each of the different models and their perturbation schemes. Initial condition perturbations can be developed in a variety of ways including Monte Carlo simulations (MCS) or lagged average forecasting ensembles (LAF). For MCS, initial conditions are perturbed through the simulations adding random coherent structures to the initial conditions (Stensrud et al., 2000). In LAF, both current model runs, and previous (hindcast) model runs are included together in an ensemble as different members. LAF is advantageous over MCS for perturbing the initial conditions as the initial conditions in the LAF are not dependent on randomly chosen errors, but instead on real error data (Kalnay, 2019). Ensembles often include several different models with different parameterization schemes. Again, this is to overcome the limitations that the parameterization schemes have as they work to encapsulate all atmospheric processes within a computationally efficient algorithm. These different models will include the various initial conditions as well, doubling the count of ensemble members. The Weather Forecast Research Team Short Range Ensemble Forecast (WFRT SREF) does not include any of these initial condition perturbation schemes though, and instead includes “best guesses” of the initial conditions using different inputs from agency run global models. More information on the WFRT SREF is included in Section 3.1. Once all the members are established, they are then averaged to get a single deterministic forecast. In almost all cases, this means they perform better than even the best members individually (Gneiting and Raftery, 2005). However, ensembles still do not create perfect forecasts. Ensembles can reduce some random errors (due to the choice of the initial conditions) and have good spread-error correlation, but are typically underdispersive (Krishnamurti et al., 2003; Raftery et al., 2005; Kang et al., 2011). The spread-error correlation measure is the day-to-day variability in the ensemble spread (the standard deviation with respect to the mean) (Whitaker and Loughe, 1998; Kalnay, 2019); and ensemble dispersion is the climatological range of the forecasting system relative to the observed outcomes (Kelsch et al., 2021). To combat the underdispersiveness of ensembles, some studies have suggested the use of weighted ensembles in which the varying model configurations are combined based on their individual performances. Typically, forecasts are improved when a weighting scheme is employed (Najafi et al., 2012). 2.2 Ensemble Size There is a large range in ensemble size (i.e., number of ensemble members) throughout the literature (Figure 1). However, most studies focused on ensembles with fewer than ten members (47%). This is interesting because Lu et al. (2007) noted that having larger ensemble sizes can help increase dispersiveness. Additionally, one article stated that regardless of the ensemble configuration, the 3 ensemble was better when six members were used instead of three (Robertson et al., 2004). This is because the inclusion of more members can help reduce the uncertainty of each individual forecast. Another study compared an optimally weighted ensemble with six high-resolution members to an equally weighted 51-member ensemble with only coarse-resolution models (Buizza, 2008). Buizza found that the 51-member ensemble performed better for 7-day forecast horizons than the 6-member ensemble over a 7-month period. That study included two severe storms with low mean sea level pressure in the Scandinavian region. This is because the 51-member ensemble had a better-tuned ensemble spread. However, the 6-member ensemble performs better before day four forecast horizons (shorter lead times). They conclude that having larger ensembles is more important for medium range forecasts compared to shorter range ones. Therefore, they recommended that ensembles contain as many accurate members as possible to ensure that forecasts can capture medium ranges (up to 7 days) if this is of interest to the forecaster, and smaller, higher resolution members only when shorter ranges are desired (less than 4 day forecast horizons). However, in this study, only perturbed initial conditions were used to differentiate the 51-members. In the case of the WFRT-SREF, members are based on the best guesses of agency model runs and are therefore prone to be more accurate than an ensemble that relies on perturbed members. Although this study found that their smaller higher resolution ensemble performed better for shorter forecast horizons, this is not in agreement with much of the literature and it is still recommended that if forecasting members are performing reasonably for the region of interest, they should be included to help reduce the uncertainty in the forecast and provide probabilistic information for forecasters. Figure 1. Histogram of the number of ensemble members used in the literature included in this review, divided into bins having a width of six. 2.3 Training Period In this thesis, training data is the forecasting and observational pairs that are used to create the weights for the ensemble. The number of training days used to construct each weighted ensemble varies heavily 4 across the literature (Figure 2). The size of the training dataset was often dependent on the amount of data available (Raferty et al., 2005). Shorter training periods allow for the system to adapt to different seasons and even weather events, and longer training periods reduce the statistical variability (Gneiting et al., 2005) and include more extreme events. This is because the use of shorter training periods would require more frequent calculations of the weights for the ensemble (i.e., on a seasonal basis) compared to longer training periods that would create weights for longer time periods (i.e., yearly) and would smooth out the seasonal variability as a result. Thus, shorter training periods yield sharper forecasts (Raferty et al., 2005). Training periods were often optimized for each study based on these principles. For example, 7 days was chosen for one study in the Pacific Northwest (PNW) (Greybush et al., 2008); however, 25 (Raferty et al., 2005) and 40 (Gneiting et al., 2005) days were chosen for other PNW studies. Both latter studies tested training periods ranging from 10 to 62 days. They found that skill increased as more training days were added, up to a certain period where it would then plateau; this point was their final chosen training period. After this plateau, skill tended to reduce as the period increased. To keep computational time short and ensure that seasonality did not affect skill, the lower end of this plateau was always taken. Further, these studies both used a sliding window technique. This technique means that the weights for each member are not based on just one select period of past days, but on a specific number of days preceding the day of the forecast (Salehi et al. 2014). This makes the weighting scheme more dynamic and responsive to seasonal (or shorter) variations. Figure 2. Histogram of the training periods used in the literature included in this review. There were other important insights about training periods that were concerned with more than just the number of training days. For example, Pagowski et al. (2005) noted that for shorter training periods using the least-squares method to create weights, there must be fewer observations than the number of ensemble members to keep the system significantly overdetermined. Lu et al. (2007) and Suh et al. (2012) suggested the use of cross validation during the training process. There are 3 different cross-validation methods (Schneider, 1997): holdout method, K-fold, and leave-one-out. Lu et al. (2007) 5 described their process as switching the training and verification periods (leave-one-out) to ensure that the training data is robust enough to correlate with the forecasts. However, very limited discussion was provided in both sources. Suh et al.’s (2012) discussion simply stated that they set aside 20 years of training data: 15 years were used to create the weights and 5 years were used for the cross validation. Based on these findings, it is recommended that the training data be adapted and optimized to future studies and applications, as in Raferty et al., (2005) and Gneiting et al. (2005). This is because different regions will have different seasonal patterns that will affect that training data dissimilarly. This will ensure the most robust and accurate data is used to create a unique weighted ensemble. 2.4 Weighted Ensemble Techniques In this section the various techniques for constructing weighted ensembles that were reviewed will be discussed and compared (Figure 3). Six techniques were found in the literature. The most popular techniques were performance-based weighting schemes (mentioned 39% of the time), which use verification metrics such as root mean squared error (RMSE), mean absolute error (MAE), mean squared error (MSE), and continuous ranked probability score (CRPS) to determine weights. This was followed by Bayesian model averaging (23%), multilinear regression (13%), clustering and likelihood functions (10%) and finally Monte Carlo like simulations (6%). Figure 3. Breakdown of the most popular techniques for creating weighted ensembles. This included Bayesian Model Averaging (BMA), Multilinear Regression, Likelihood Functions, Clustering, Monte Carlo Simulations, and Performance Based weighting. The latter is broken down further into weighting based on root mean square error (RMSE), the continuous ranked probability score (CRPS), mean square error (MSE) and mean absolute error (MAE). 6 2.4.1 Performance-Based Weighting (PBW) The performance-based weighting schemes were the most popular as they are easy to calculate, interpret, computationally efficient, and based on metrics that have been well studied and documented in the field of meteorology (RMSE, MAE, MSE, and CRPS). One of the most popular ways to create weights in this way is to use inverse-error metrics (Greybush et al., 2008). This ensures that the weights are directly related to the magnitude of each member’s error. This means that the members with the largest error will be given the smallest weights and the models with the smallest error will be given the largest weights. Gensler and Sick (2016) expanded upon this inverse weight function by adding an exponential function to emphasize the better performing models and arbitrary coefficients to ensure stability in the functions. They applied this function to both an ensemble of weather models and wind turbine power models that were combined into their ensemble in their “cooperative approach”. These results were then compared to “state-of-the-art" ensemble approaches such as k-nearest neighbor (k-NN), artificial neural networks (ANN) and linear regression (LR). The authors found that the ANN approach had the lowest RMSE, but all the ensemble techniques performed quite similarly. The cooperative approach did not yield lower RMSE than the state-of-the-art methods; however, it was computationally inexpensive to both create the model and create the weights, it did not depend on the number of ensemble members, and it is modular in that calculations happen independently and allow for the number of predictors to be controlled depending on the computer load. In Buizza (2008), the weights were a function of the forecast RMSE compared to the other members and the percentage of the observed anomaly (with respect to climatology) that is projected into the forecast anomaly. The reason for this two-stage approach is that it is more general and ensures that the weights are not optimized around a single value that may not be robust. They also chose to use a Euclidean norm to normalize the vectors and compute the vector scalar products to ensure consistency. However, this method makes the weights specific to the region and variables of interest and makes it unfit for transfer between different regions as climatology plays a role in the weights. They also found that the weights only needed to be computed once as the weights had low sensitivity to the time-frame picked. Computing the weights only once though for each region does significantly reduce the computations. The weakness of this one computation approach though is that it means that the weighting technique may not account for seasonal performance variability amongst the members. There is a possibility that this variability simply did not exist for their study, but it was not discussed in their work. When using error as the performance-based metric on a combined measurement of 16 different variables (i.e., precipitation, temperature, wind components, etc.), no improvement of the weighted ensemble over simple averaging of the members was found unless they were examining short periods of time (Du and Zhou, 2011). They also found that the weighted ensemble had no impact on random errors or systematic error compared to the equally weighted ensemble. This led them to conclude that the improvement of a variably weighted ensemble over an equally weighted ensemble may be limited by size, and that larger ensembles will typically see less improvement when weighted compared to smaller ones. Further, their study looked at the accuracy of weighted ensembles to predict spatial structures and extreme values. From their analysis, they found that studies not concerned with spatial structures and extreme values can get good performance using an equally weighted ensemble. However, if one were interested in spatial details and extreme values, they suggest using the best high-resolution 7 member. However, one should take caution when doing this and should make sure it is better than the ensemble mean before doing this. Some studies have taken the minimization of errors much further and applied machine learning to their algorithms to create weights that result in the best performing forecast (Amarasinghe et al., 2020). These optimization schemes can be constrained or unconstrained in which the weights are forced to be positive or can take any integer value (positive or negative), respectively (Guerra et al., 2020). In using these optimization schemes, various error calculations or skill scores can be utilized. For example, Guerra et al. (2020) used the Brier score, MAE, correlation coefficient, rank correlation coefficient, reliability, true skill statistic, Heidke skill score, accuracy, critical success index and Gilbert skill score to calculate different sets of scores. They found that the best performing ensembles were constructed with the Brier score and correlation coefficient. There were also several studies that examined various metrics for creating weights and then compared them against each other. Sofiati and Nurlatifah (2019) used RMSE and a ratio of the hit rate to false alarm ratio to determine weights. However, they found both the variably weighted and equal-weighted ensembles produced similar results for precipitation forecasts. Other authors preferred to combine different verification metrics together to make the weights more robust and based on several attributes of the ensemble members rather than just the error. This was done by Suh et al. (2012) who combined bias, RMSE, absolute correlation and original correlation in various configurations to create weights for precipitation and temperature forecasting members. They found that this technique did not perform as well as the ensemble that used multilinear regression (see section 3.3.3) to create weights when the bias and RMSE of both ensemble configurations were compared. Other authors chose to establish different weighting schemes for different forecast types. In Gneiting et al. (2005), both MAE and MSE were used to create weights for deterministic members; and the CRPS and ignorance score were used to create weights for probability density function forecasts. Here, 48-hour forecasts of temperature and mean sea level pressure were considered. All 28 ensembles they constructed had lower RMSE and MAE than the members themselves, and this was true for the equal-weighted ensemble, or the simple bias corrected equally weighted ensemble. Liu et al. (2017) based their weights on the bias and the Nash-Sutcliffe efficiency of each flood forecasting ensemble member. However, they also introduced an additional step into their weighting schemes in which they examined the member threshold (based on climatology) information in combination with the forecasted mean and member means alone. This was to investigate the importance of the full spectrum of information from each member. As expected, the ensembles that were constructed using all the member information and not just the means had better reliability, sharpness, and resolution (determined through the Brier skill score, CRPS, rank histograms and reliability diagrams). Another interesting performance technique is that of Stanger et al. (2019). In their method they used perturbed wind speed forecasts to create an ensemble forecast of wind power data. They found that their weighted ensemble performed similarly to that of an equally weighted ensemble, probably due to a less intensive weighting scheme. However, both ensemble setups did outperform climatology and the 8 individual members, as expected. To improve this approach, they recommend that weights be adjusted to the weather regime to get optimal results. In summation, there are various paths that can be taken when investigating performance-based techniques for weighting ensemble members. However, there is contention around the utility of these methods as they appear to not always outperform an equally weighted ensemble. Typically, when climatology is included in the weighting scheme though, better results are achieved. Methods that can include climatology, remain simple and do not try to overcomplicate the process remain the preferred weighting schemes in performance-based weighting. These are still not always successful though and have created a space for other weighting techniques to be established that will be discussed in the following sections. Climatology could not be included in this study due to a lack of data. However, if possible, it should be considered in future studies. 2.4.2 Bayesian Model Averaging Bayesian Model Averaging (BMA) is the weighted average of probability density functions centered on the individual bias-corrected forecasts. These weights are equivalent to the posterior probabilities of the models generating the forecasts and the model’s relative contribution to the predictive skill of the training period. In essence, these weights indicate the usefulness of the ensemble members. The BMA predictive variance can be decomposed into the between forecast variability and the within forecast variability. Often, the latter is not included in the probability density function which is why BMA is good at spread-error relationships but not dispersion. BMA is especially popular in hydrological forecasting as it is easy to implement for these types of systems (Bradley et al., 2015). This is because it does not require calibrations with past data and is able to adapt to the relationship between climate and specific variables. In some articles, BMA is used solely for determining the uncertainties in hydrological models (Zhang et al., 2020). However, most authors find it useful for creating weights (Kim et al. 1006; Duan et al., 2007; and Raftery et al. 2005). For purely weather applications, Lee et al. (2012) found that their 24-member ensemble of 48hr forecasts of 2m-temperatures and 10m-wind speeds had improved ensemble dispersion and performance after using BMA to weight the members. Further, Raferty et al. (2005) found that their ensemble of 48hr temperature forecasting members weighted using BMA had a 7% reduction in RMSE compared to the best ensemble members; an 8% lower RMSE than the equally weighted ensemble; and a 56% reduced RMSE compared to climatology. Noticeable weaknesses of the Bayesian schemes are that they have a tendency toward higher precipitation weights for high latitude regions, for global climate models, but this can be rectified by averaging the weights across different subsamples and spatial regions (Roberston et al., 2004). In addition, since BMA cannot converge to a daily maximum, it is sensitive to the starting values, and one needs to ensure that the starting values are based on past values (Raferty et al., 2005). Further, other studies have found that BMA constricts ensembles in a way that overestimates the probability of extreme events if the ensemble mean differs significantly from the climate mean (Bishop and Shanley, 2008). To combat these deficiencies, Bishop and Shanley (2008) proposed the use of BMA along with Baye’s theorem and an estimate the climatological distribution to tune the likelihood distribution into a posterior distribution, given the current ensemble mean. Using their idealized randomly generated 16- 9 member ensembles, they proved that this process was successful in forecasting both extreme and non-extreme weather. Berrocal et al. (2007) opted to use a specific spatial-BMA approach that includes flow-dependent information of the weather field, so that dynamical information is not lost in the statistical process. Their results indicate that this new spatial-BMA outperformed both regular BMA and another approach, the geostatistical output perturbation. The spatial BMA was also better than the raw ensemble. Spatial-BMA is especially good for variables that approximate a Gaussian distribution (i.e., sea level pressure). However, to make this method robust, it requires bias correction (Berrocal et al., 2007). This is because the current method is linear and does not account for altitude, land use, longitude, or distance to the ocean. They note that it is typically recommended to use a method like nearest neighbor for bias correction or simple additive bias correction. However, they chose to use simple linear bias correction that does not require extensive data such as altitude, land use, or region type (i.e., marine, continental, etc.) due to data and computational constraints. Like performance-based metrics, there is a lot of contention surrounding the expected performance of BMA created weights compared to equally weighted ensembles or other methods. However, if BMA is used in conjunction with other post-processing techniques, more utility can be realized, and the forecasts can undergo further improvements. Most of the applications in the literature, though, were for climatological or hydrological applications, indicating that BMA may be better suited to longer-term or more idealized forecasting. Berrocal et al. (2007) note that BMA is well suited to approximate Gaussian Distributions, which are not often found in shorter-term weather forecasting. 2.4.3 Multilinear Regression (MLR) MLR creates weights to achieve the least squares error between the member forecasts and observations. Least-square regression creates a set of linear equations that is then solved for the regression coefficients that become the weights. These weights can be positive or negative (Raferty et al., 2005). Multilinear regression is often popular as it does not require parameters (rules that aid in model learning) to be calculated and it is computationally inexpensive (Thorey, et al., 2018). However, Thorey et al. (2018) found that the method is good for events of low probability, but overpredicts events with high probabilities. Further, Lu et al. (2007) found that a weighted ensemble using MLR does not perform well in the upper troposphere compared to the individual deterministic forecasts. This is because of the small smoothing effects of the ensemble that is overwhelmed by the large wind speed and temperature variations at jet-stream levels. Suh et al. (2012) compared PBW schemes, MLR and equal weights for a range of variables including temperature, precipitation, and atmospheric pressure. They found that the MLR ensemble performed worse than PBW, but both MLR and PBW performed better than the equally weighted ensemble. Lu et al. (2007) also investigated MLR, however they compared it to only a time-lagged equally weighted ensemble, and they only investigated the first 3 hours of the forecasts. Their results indicated that for this 3-hour period, MLR did outperform the time-lagged equally weighted ensemble. Further, Gensler and Sick (2008) found that MLR required a lot more training data. This is because MLR will force extrapolation of data when given data outside of the training dataset. However, when given a large enough training dataset, MLR becomes independent of the amount of training data and will no longer 10 be forced to extrapolate (Gensler and Sick, 2016). However, no mention is made about what a “large” training dataset is. This method has also been the least studied compared to both PBW and BMA, so caution should be exercised when implementing this method before its utility over the other, better studied methods is known. 2.4.4 Clustering Since BMA and performance-based metrics performed only slightly better than an equally weighted scheme, Greybush et al. (2008) explored the utility of clustering schemes for weighting ensembles that forecast surface temperature in the Pacific Northwest. They chose clustering because it provides an array of temporally non-continuous data that allows for the minimization of MAE. They identified two different clustering methods: K-means and a genetic algorithm. When using the K-means methods they found that the forecast skill improved as the cluster number increased to 24, then the skill started to decline after that due to overfitting. For the genetic algorithm approach, they found that that it was also better with fewer clusters; however, it was very computationally demanding compared to k-means and was just “ok” for real datasets. As a result, the K-means approach was regarded as the best choice for clustering. However, they note that future studies should confirm these results using larger datasets and in different regions. Salehi et al. (2014) also found success with using clusters to establish weighting schemes for temperature and relative humidity forecasts in the tropical Pacific. Their focus was specifically on local and global anomaly detection in forecasts. They compared their weighted ensemble to an outlier detection technique: incremental local outlier factor (iLOF). Using an ROC curve, they showed that their ensemble configuration was much better suited to outlier detection compared to the iLOF algorithm. Other authors (Naiafi et al., 2012) also investigated the use of fuzzy K-means clustering on streamflow forecasting members to create weights. They found the resulting ensemble performed similarly to other techniques (BMA) but that they all focused on different strengths of forecasts. Therefore, their recommendation is for the merger of various weighted and non-weighted techniques to construct ensembles. This will allow for ensemble systems to better account for uncertainties and make better predictions. In other cases, clusters are used as an initialization step to identify different regions in which different weights should be assigned (Amarasinghe et al., 2019). That is, some members will perform differently in different regions depending on the skill of the model. For example, a higher-resolution model may perform better in mountainous terrain compared to a courser model. In summation, clustering was introduced as a weighting scheme to add strength to already existing methods. It is a good way of identifying different regions or different weather structures (i.e., anomalies) in which different weighting schemes can be applied. However, there is additional computational burden of adding clustering to weighting schemes. When introducing clustering, it is important to consider how much utility will be added by weighting different spatial regions separately, and the cost of computations that it adds. 11 2.4.5 Likelihood Functions (LF) Doycheva et al. (2017) used the generalized likelihood uncertainty estimation (GLUE) to create weights for precipitation forecasting members. This method quantifies the uncertainty in a hydrological model based on the assumption that different model initializations result in equally good model performances. It required manual pairing of radar and forecasting images by an expert for the training dataset, after which the rest of the data was able to be automatically matched. This study calculated different weights for different forecast lead times. They found that the weighted ensemble reduces the uncertainty, eases analysis tasks of forecasters, and improves ensemble system performance. However, the weights were not flexible and were not able to account for model performance changing over time due to seasonal variations. Other authors chose to determine weights by maximizing the log likelihood. Robertson et al. (2004) had initially used the log likelihood to enhance their BMA weighting scheme for precipitation forecasting. There results indicated that the addition of the likelihood technique increased the ranked probability skill score for the ensembles. This approach was an improvement over that of Rajagoplan et al. (2002) in that they reduced dimensionality by averaging across all the subsamples and spatially smoothing the likelihood. Najafi et al. (2012) compared likelihood methods to clustering methods and did not find significant differences between the performances of the ensembles predicting streamflow. Although this method appears to be very useful, the requirement for manual pairing of data points makes it difficult to use operationally. Additionally, the method is best used in conjunction with other methods, which increases the computational requirements for the weighting process. There is promise for this technique in the future once it has been refined further. However, it is currently too cumbersome for frequent use in current meteorological applications. 2.4.6 Monte Carlo Another popular weighting scheme utilizes the Monte Carlo method (Patra et al., 2013). This method approximates the state of the probability-density-function evolution. It is computationally taxing, making it unfeasible for high-dimensional dynamical systems (Patra et al., 2013). Thus, Guerra et al. (2015) chose to use Monte Carlo like simulations that apply a decision threshold to the combined probabilities, to maximize the Heidke Skill Score (HSS) for solar flare forecasts. They found that this weighted scheme performed similarly to an equal-weighted ensemble with the same members over a variety of skill metrics (Brier skill score, reliability, resolution, ROC curve, MAE, etc.). However, this may have been because of the small domain size they used. Due to the computational cost of this method and relatively minimal applications in the field, it would not be a suitable option for running operationally at this time. There may be promise for this technique in the future once it has been further refined. 2.5 Summary of Literature Review In this section, the methodologies from several studies on establishing weighted ensembles for weather forecasts were discussed. Both the number of training days and ensemble members used in the ensemble configuration are as important as the weighting scheme itself. In general, weighted ensembles should include enough 12 training days to encompass most of the range of possible values in the forecast. Also, the training dataset should be able to account for seasonality by weighing each season separately. Therefore, the optimal amount of training data needs to be deduced for each region individually. The number of ensemble members should also be sufficiently large (based on the acceptable user error) (Milinski et al., 2020). There is no theoretical upper limit on this number, but this will often be capped by the computing power of the ensemble host itself and data resources. Having many ensemble members ensures that the ensemble is dispersive and has minimal random biases. In terms of weighting methods, several different types were discussed including PBW, BMA, MLR, clustering, likelihood functions, and Monte Carlo schemes. The performance-based techniques are the most popular, consistent, and easy to interpret. However, they often only minimally increase the ensemble skill, if at all. However, these studies typically used fewer than 10 members, which perhaps does not allow for a large enough sample size to test. BMA is also quite popular, but it is better suited to more idealized data, such as the distributions found in climatological or hydrological forecasts, rather than short-term weather forecasting. For clustering, likelihood functions, and Monte Carlo schemes, they are all able to add skill to the forecast. However, this comes at a great computation cost. Further, likelihood functions need manual pairing, and clustering can only do part of the weighting. Therefore, we recommend the use of PBW for operational weighting schemes and choose it as our method for this study. This is because we have 51 members and may be able to see a greater increase in skill than studies with 10 ensemble members. Further, it is computationally inexpensive and was able to outperform BMA and MLR. Further, PBW is well understood in scientific literature and easily interpreted. Due to the limited amount of literature in this area, anyone establishing a weighted ensemble should not limit themselves to one technique. Although conclusions were drawn for the ensemble and region tested in this study, one might find different conclusions in different regions. Meanwhile, newer research is being conducted on neural networks or deep-learning which might have great utility in the future (Amarasinghe et al., 2020; Weyn et al., 2021). 13 3. Methodology In this section we describe the procedure for creating the weighted ensemble, we list the ensemble members used and discuss the verifying datasets. All work conducted for this report is documented on GitHub and can be found in the following repository: https://github.com/rrmckinney/verif-post-process/tree/main/weights. 3.1 Datasets Two years of ensemble-member forecasts and observational data were collected from October 2021 – October 2023 for this analysis. More information about these forecasts and observations is found below. This study focuses solely on the 60 hr outlook forecasts from each member, as it is the only outlook horizon that contains all 51 members. Further, only 2m-temperature (Kalman Filtered; KF), 10m-wind speed (KF) and hourly precipitation will be considered here. 3.1.1 Ensemble Members We use all 51-members of the Weather Forecasting Research Team (WFRT) 00Z short range ensemble forecast (SREF). The members consist of six distinct model configurations and eight distinct initial condition sources. This results in a range of domains with horizontal grid sizes ranging from 1 km to 108 km, nested over southwestern British Columbia (BC). Figure 4 shows this nesting relationship. The six models consist of the Model for Prediction Across Scales (MPAS 7.0; Skamarock et al., 2012); fifth-Generation Penn State/NCAR Mesoscale Model (MM5; Grell et al., 1994); and four versions of the Weather Research and Forecasting Model (WRF-NMM 3.6.1 and WRF-ARW 3.7.1, 3.9, 4.2.1; Skamarock et al., 2008; Skamarock et al., 2019). The eight initial condition schemes consist of input from the 0.5° action de recherche petite echelle grande echelle integrated forecast system (ARPEGE IFS; Déqué et al., 1994); 15 km Global Environmental Multiscale Model (GEM; Côté et al., 1998); 0.25° Global Forecast System (GFS; NCEP, 2023a); 0.5° Global Forecast System (GFS; NCEP, 2023a); 0.25° Icosahedral Nonhydrostatic Model (ICON; Zängl et al., 2015); 0.5° Navy Global Environment Model (NAVGEM; Hogan et al., 2014); 32 km North American Mesoscale Model (NAM; NCEP, 2023b); and the 10 km Regional Deterministic Prediction System (RDPS; Caron et al., 2016). Detailed information regarding each of the members is found in Table 1 including the variables of each forecast, domain sizes and forecast outlooks. Forecast output is saved for every hour of the forecast horizon. For more information on the WFRT ensemble suite please refer to: https://weather.eos.ubc.ca/wxfcst/html-etc/model-metadata/summary.html. 14 Figure 4. Map of all the domains included in the WFRT SREF. Table 1. List of WFRT SREF Member Attributes. MODEL INITIALIZATION SOURCE HORIZONTAL GRID SPACING (KM) FORECAST HORIZON FOR EACH GRID SPACING (HR) MPAS 7.0 GFS 0.25° 25, 75 120, 120 MM5 NAM 32 km 4, 12, 36 60, 60, 60 MM5 GFS 0.5° 1.3, 4, 12, 36 84, 84, 84, 84 WRF-ARW 4.2.1 NAM 32 km 4, 12, 36 60, 84, 84 WRF-ARW 3.7.1 NAM 32 km 4, 12, 16 60, 84, 84 WRF-ARW 3.7.1 GFS 0.5° 1.3, 4, 12, 36 84, 84, 180, 180 WRF-ARW 4.1.2 ARPEGE 0.5° 9, 27 102, 102 WRF-ARW 4.1.2 GFS 0.25° 9, 27 180, 180 WRF-ARW 4.1.2 ICON 0.25° 9, 27 180, 180 WRF-ARW 4.1.2 RDPS 10 km 1, 3, 9 84, 84, 84 WRF-ARW 3.7.1 GEM 15 km 4, 12, 36 84, 168, 168 WRF-ARW 3.7.1 NAVGEM 0.5 4, 12, 36 84,84, 84 WRF-NMM 3.6.1 NAM 32 km 4, 12, 36 84, 84, 84 WRF-NMM 3.6.1 GFS 0.5° 12, 36 180, 180 WRF-ARW 3.9 NAM 32 km 9, 27, 81 84, 84, 84 WRF-ARW, 3.9 NAM 32 km 12, 36, 108 84, 84, 84 WRF-ARW 3.9 GFS 0.5° 12, 36, 108 180, 180, 180 WRF-ARW 3.9 GFS 0.5° 9, 27, 81 180, 180, 180 15 3.1.1 Observations The hourly observational data is collected from 90 weather stations in southwestern (BC). These stations are owned and maintained by a variety of agencies including BC Hydro, Coastal Mountain Bus Company (CMBC), Environment and Climate Change Canada (ECCC), Greater Vancouver Regional District (GVRD), NAV Canada, National Centers for Environmental Prediction (NCEP), and the WFRT at the University of British Columbia. Figure 5 shows a map of all these stations. Figure 5. Map of all the observation stations for this study. A quality check was conducted on all two years of station data that included removing any station from a forecast verification that was missing more than 40% of data from a given week and any data that fell outside of the Canadian extremes record without proper documentation. This excluded 32 stations. Canadian extreme weather was chosen as the threshold for the weather variables as we wanted to make the threshold large enough that they did not discount data outside the climatological norm for this region in the case extreme weather did occur. Table 2 describes the temperature, wind speed and precipitation records used here. Table 2. Threshold Values used for Quality Check on Observation Data. VARIABLE THRESHOLD LOCATION OF RECORD SOURCE PRECIPITATION 250 mm/hr Buffalo Gap, SK 1961 (Canada, 2013) WIND SPEED 400 km/hr Edmonton, AB 1987 (ECCC, 2017) TEMPERATURE MINIMUM -63°C Snag, YT 1947 (ECCC, 2017) TEMPERATURE MAXIMUM 49.6°C Lytton, BC 2021 (Canada, 2021) 3.2 Weighting Technique Weighting was done using a variety of methods to both ensure that this process could be run operationally and to ensure that the most optimized weights were being produced. In the literature, most studies concluded that shorter training periods for weights performed better as they were better able to adapt to seasonality. However, no longer term schemes (greater than a year) investigated using seasonal data from the past to construct weights. 16 In this study, to overcome the seasonality shortcomings of these studies with longer training periods, we adopted a hybrid approach in which weights were created using the previous year of data (October 2021-September 2022) but breaking the weights into the four meteorological seasons. Since the dataset began in October, but the meteorological season of Fall includes September, September 2022 was tacked onto October 2021-November 2021 to create the weights for that season. Three different weighting functions we employed using hourly observational-forecast data pairs: inverse of the error metric, value based logistic function and rank based logistic function. All are described in more detail below. The second year of data (October 2022 – September 2023) was used as an independent test of the weighting schemes. A schematic of how this weighting scheme was implemented is provided in Figure 6. Figure 6. Schematic of seasonal weighting scheme. We also wanted to test the idea of sliding windows (Gneiting et al., 2005; Raferty et al., 2005) with shorter amounts of training data. For this, the performance of each member from the previous week or month was used to rank the members and produce the weights using the rank based logistic function which will be described in more detail below. Only the second year of data (October 2022-September 2023) was used for this technique so it could be compared to the seasonal based weighting. A schematic of how this weighting scheme was implemented is provided in Figure 7. Figure 7. Schematic of sliding window weighting scheme. For all weighting schemes, different weights were made for each meteorological season and all variables based on the following continuous verification metrics: mean absolute error (MAE), root mean squared error (RMSE) and Spearman rank correlation (SRC). Five additional weights were also calculated for wind speed and precipitation based only on the following categorical statistics: probability of detection (POD), probability of false detection (POFD), Pierce skill score (PSS), Gilbert skill score (GSS) and the critical success index (CSI). Equations for all 8 verification metrics can be found in Appendix B. 17 Various verification metrics were chosen to calculate the weights as they all have different strengths. Both RMSE and MAE measure accuracy and ignore linear association (Murphy, 1988; Stensrud and Wandishin, 2000). However, RMSE is better at penalizing outliers (Brooks and Doswell, 1996). To measure linear association, correlation must be calculated. We chose SRC as it is like the well-known Pearson correlation but does not assume the data is normally distributed as it uses ranks instead of actual values (Schober et al., 2018). SRC is not able to be calculated though for data with zeros, which are very common in the precipitation dataset (which will be seen in the results). For categorical scores POD and POFD were chosen as they are simple metrics of forecast performance that are easy to interpret and are the foundation of most of the other scores. PSS is regarded as the most robust (Stephenson 2000; Jacobs, 2005) and the only truly equitable (Hogan et al., 2010) categorical skill score. We choose to compare it here with the GSS (or Equitable Threat Score) as although it has been regarded as inequitable and encourages hedging (Pasaric and Juras, 2011), it has been determined to be an effective measure of ensembled member quality (Gebhardt et al., 2011). Lastly, the CSI was chosen as it depicts the forecasts’ ability to forecast events that were observed. It has been noted though, that is does a poor job of distinguishing the source of error in the forecast (either from false alarms or misses). Therefore, although we hope to get insight from all of these scores, PSS will be examined the closest as it is the most well regarded. Verification scores to construct the weights were made by comparing each of the forecasting members to the observations for every hour of the 60 hr forecasts at each station location to ensure weights reflected the members ability to resolve different subregions within the study domain. For each weighting scheme, this resulted in 12 weights for temperature and 36 weights for wind speed and precipitation, each. To compare all three variables candidly in this report, only the continuous-score oriented weights will be included in full analysis here. Though, short comments will be made about the other categorical scores. 3.2.1 Logistic Function The first weighting scheme was based on the logistic function (LF). Two different variations of it were employed based on: 1. actual verification score values of each member (VLF); and 2. ranks of each member (RLF) based on their verification scores. For the first variation, VLF, the logistic function is given as: 𝑤𝑉𝐿𝐹 = 𝐿1+𝑒−𝑘(𝑥−𝑥𝑜) Eq. 1 where 𝑤 is the resulting weight, 𝐿 is the maximum value weights can take [1], 𝑘 is the steepness parameter, 𝑥 is the real value (either the rank or verification score for each model) and 𝑥𝑜 is the midpoint of all 𝑥 . After the weights were calculated they were all normalized so that they added to 1. Several different steepness parameters were investigated to determine the sensitivity of the function. For the second variation, RLF, the logistic function is given as: 𝑓 = 𝑒−6⋅(𝑟−𝑟ℎ𝑎𝑙𝑓)/(𝑟𝑐𝑢𝑡−𝑟ℎ𝑎𝑙𝑓) Eq. 2a 18 𝑤𝑅𝐿𝐹 = 𝑓1+𝑓 Eq. 2b where 𝑟 is the rank of each member, 𝑟ℎ𝑎𝑙𝑓 is the rank that signifies halfway on the logistic curve, and only members whose rank is greater than 𝑟𝑐𝑢𝑡 are included. After the weights were calculated they were all normalized so that they added to 1. For both 𝑟𝑐𝑢𝑡 and 𝑟ℎ𝑎𝑙𝑓, several values were iterated through to test the sensitivity of the curve to each. Figure 8 provides a schematic of both of these parameters. Figure 8. Schematic of the Rank Based Logistic Function (RLF) parameters, rcut and rhalf. 3.2.2 Inverse of the Error Metric (IEM) This scheme (IEM) is based on the work by Greybush et al. (2008) and was based solely on the verification metric result real value. Weights here are given by: 𝑤𝐼𝐸𝑀 = 𝑝𝑖−1∑ 𝑝𝑗−1𝑛𝑗=1 Eq. 3 where 𝑤 is the resulting weight, 𝑝𝑖 is the verification error metric for member 𝑖 out of 𝑛 members. After the weights were calculated they were all normalized so that they added to 1. 3.2.3 Variable Number of Members (VNE) There was also interest in how the number of ensemble members affected the overall skill of the ensemble. Here, no weighting schemes are used, instead the models were ranked based on the nine different verification scores for each variable. Then the ranks for each variable were added together for each member to get an overall rank. Using these ranks, an ensemble was constructed using the top member, then the top two members and so forth until all 51 members were used. 3.2.4 Significance testing To test the significance of the weighted ensemble compared to the SREF, a T-Test was employed on verification statistics (MAE, RMSE, Spearman Rank Correlation) of the weighted ensembles and SREF. This was done to illustrate the significance of the weighted ensemble’s ability to reduce errors compared to the SREF. Results of this are presented in the next section. 19 4. Results In this section results will be presented for the calculated weights, the performance of each weighted ensemble across the study spatial and temporal domains, the statistical significance of our results and the importance of the number of ensemble members. 4.1 Weighted Ensembles 4.1.1 Weights In this section, a breakdown of the weights calculated for all of the seasonal weight schemes will be presented. The sliding window cases are excluded here as those weights change every week or month. They follow the RLF-rcut=30 calculation scheme though and their distribution follows the same pattern, though changing weekly or monthly, respectively. Firstly, we will discuss the effect of the k parameter on the VLF weights. k-parameters of 40, 80, 100, 150, 200, 500, and 1000 were chosen to test the sensitivity of the scheme to the k-parameter. It was noticed that k-parameters greater than 200 often lead to mathematical errors and were not able to be further investigated. It was also noted that before normalization, weights were variably different between members; however, after normalizing the weights, they became quite similar between members due to the great reduction in size of the weights so that they add to one. Figure 9 shows the fall weights after normalization for the VLF for all three parameters based on MAE verification of temperature. The highest weights are awarded to the MPAS members followed by the RDPS members. The lowest weights are awarded to the MM5 (NAM) and MM5G (GFS) members. For the RLF, weights retained their differences even after normalization, which provided a much more interesting curve of members to be included in the weighted ensemble as opposed to the VLF. A breakdown of the weights for the RLF for temperature are shown in Figure 10 and Figure 11. 20 Figure 9. Values-Based Logistic Function (VLF) calculated weights for temperature based on the MAE verification metric. The effect of different k-parameters (40, 100, 200) is shown. Weights are specific to fall. Figure 10. Rank-Based Logistic Function (RLF) calculated weights for temperature. rcut and rhalf values are set to 30 and 16, respectively. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. Narrow lines indicate near zero weights. 21 Figure 11. Rank-Based Logistic Function (RLF) calculated weights for temperature. rcut and rhalf values are set to 15 and 7, respectively. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. Narrow lines indicate near zero weights. The highest weights are awarded to MPAS and RDPS members again. Due to the scheme calculation, many of the members have near zero weights, including the MM5 (NAM) and MM5G (GFS) members. These also had the lowest weights in the VLF schemes. Similar results from the MAE and RMSE indicates limited outliers in the dataset, as RMSE would score outliers harsher than MAE. However, since there are limited outliers in the members dataset, the RMSE scores similarly to MAE. Therefore, in future parts of this analysis, only MAE will be considered. The IEM weights were similarly nondiverse as the VLF. Model weights all fell between 0.015 and 0.025. This is shown in Figure 12. Here the highest weights were again given to the MPAS and WFR3 (RDPS) members (highest resolution). However, the lowest weights were given to the WRF3 (GFS) members and the lower resolution RDPS member. This was also the case for precipitation and wind speed weights made with MAE, RMSE, and SRC as well as for POD, POFD and CSI. More diverse weights were created though when PSS and GSS were used as shown in Figure 13 for precipitation. There was disagreement between these three verification schemes on the best performing members. PSS had WRF3 (GFS), WRF3 (GEM) and RDPS as the highest performers and WRF4 (NAM), MM5 (NAM) and MPAS to be the weakest performers. In contrast, CSI and GSS weighting schemes had WRF4NAM and WRF3 (RDPS; high-resolution) as the highest performers and WRF3 (GFS), WRF3 (GEM) and WRF4 (RDPS; low-resolution) to be the weakest performers. This is more consistent with the weights calculated for the RLF schemes. 22 Figure 12. Inverse Error Metric (IEM) calculated weights for temperature. Weights were calculated using mean absolute error (MAE), root mean squared error (RMSE) and spearman rank correlation (SRC), respectively. All weights are specific to the season of fall. Figure 13. Inverse Error Metric (IEM) calculated weights for precipitation. Weights were calculated using Pierce Skill Score (PSS), Critical Success Index (CSI) and Gilbert Skill Score (GSS) respectively. All are specific to the season of fall. 23 4.1.2 Weighted Ensemble Forecasts 4.1.2.1 Spatial Results A subset of the weighted ensembles constructed will be presented here. This subset will contain a set of non-diverse weights (VLF-k=100 - MAE) alongside the more diverse weights: RLF-rcut=30(MAE), SW-weekly (MAE), and IEM(PSS) to get a comparison of how these differing weight results affect the ensemble weekly forecast. First, we will examine the spatial results of the weighted ensemble weekly forecasts compared to the equally-averaged SREF for temperature (Figure 14). Only VLF-k=100(MAE), RLF-rcut=30(MAE) and SW-weekly (MAE) will be compared here as weights made with categorical scores were only constructed for precipitation and wind speed (see section 3.2). Figure 14. Spatial comparison of temperature forecasting performance of the equally weighted short-range ensemble forecast (SREF) and three weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; right); and the Sliding Window-weekly (SWeekly; bottom). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. 24 The results of Figure 14 indicate that the weighted ensembles perform worse on average than the SREF. These differences are quite small though (<1°C). It is interesting to note that the weighting scheme with more diverse weights (RLF-rcut=30) and thus, most nuanced than the equally weighted ensemble produces lesser differences in MAE compared to the equally weighted ensemble; whereas the weighted ensemble with non-diverse weights (VLF-k=100) performs much differently than the equally weighted cases as denoted by the stronger colors in the image. The SW-weekly approach alarmingly indicates much worse weighted ensemble performance compared to the SREF as shown by the large number of dark blue dots. This is unexpected as the literature championed the SW approach as being the most accurate as it accounts for sub-seasonal variations. Next, we will compare the weighted ensembles spatially for wind speed (Figure 15). In this comparison, we also include a weighted ensemble constructed with a categorical skill score: IEM(PSS). Figure 15. Spatial comparison of wind speed forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; right); the Sliding Window-weekly (SWeekly; bottom left); and the Inverse Error Metric (IEM; bottom right). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. 25 For wind speed forecasting there is an overwhelming agreement between stations that on average, the weighted ensemble, RLF-rcut=30, performs better than the SREF. However, this performance is only 1 km/hr better than the SREF which is not a large difference for wind speed forecasting. The VLF-k-100 ensemble also indicated better performance than the SREF; however, this difference is even smaller than 1 km/hr. It is inconclusive as to whether the SW-weekly performs better than the SREF. We find little agreement between the stations, with some indicating better performance for the SREF (Vancouver Island) and others indicating better performance for the weighted ensemble (Lower Mainland). Finally, the IEM performs quite similarly to the SREF on average aside from a few stations where the IEM performs slightly better (<1 km/hr) than the SREF at the tip Vancouver Island. Lastly, we will compare the weighted ensembles spatially for precipitation (Figure 16). Figure 16. Spatial comparison of precipitation forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=100 (VLFk100; top left); the Rank Based Logistic Function for rcut=30 (RLFrcut30; top right); the Sliding Window-weekly (SWeekly; bottom left); and the Inverse Error Metric (IEM; bottom right). The dots indicate the station mean absolute error (MAE) mean difference between the SREF and weighted ensembles throughout the validation period (October 2022-September 2023). Dots with blue color indicate the weighted ensembles have a higher MAE than the SREF and dots with red color indicate the SREF having a higher MAE than the weighted ensembles. 26 The scale here is much smaller than the temperature and wind speed cases as the error for precipitation is much less. This is due to the number of days with no precipitation in the dataset skewing the errors. We chose to include all zero days of precipitation in our results, as it helps us indicate false alarms in the dataset. In all four cases above, the weighted ensembles outperform the SREF at most stations. This is especially true for the RLF-rcut=30 case where the red dots are the darkest, followed by the SW-weekly approach and then the IEM case. These differences though are very small and amount to much less than the minimum sampling amount of the standard tipping rain bucket (0.2mm). 4.1.2.1 Temporal Results Here we will examine the results for each weighted ensemble temporally. In Figure 17, verification metrics for each weighted ensemble are compared altogether for temperature for all of the forecasts during the study period. Figure 17. Temporal comparison of temperature forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-weekly; SW-monthly); and the Inverse Error Metric (IEM). Above, we can see that in almost all cases, all of the weighted ensembles and the SREF perform quite similarly, with MAE and RMSE ranging from near 0 to 6°C; and SRC ranging from -0.4 to near 1. As we witnessed before in the weighting, MAE and RMSE verify all of the members similarly. This again attests to the limited outliers in the dataset, which is a function of the averaged ensembles reducing outliers in each members’ forecasts. SREF has the lowest MAE and RMSE on average, but also the lowest SRC. 27 However, these differences are rarely greater that 1°C, and there is no set pattern to the differences (i.e. no one member always with lower error or higher correlation). Next, we will investigate the temporal performance of wind speed (Figure 18). Figure 18. Temporal comparison of wind speed forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-weekly; SW-monthly); and the Inverse Error Metric (IEM). For wind speed we see a similar result to temperature with all of the ensembles verifying around the same ranges. Here MAE and RMSE range from 0 to 17.5 km/hr; and SRC ranges from -0.75 to 0.6. The highest MAE and RMSE go to the SREF most commonly, whereas the lowest SRC is often award to SW-weekly and RLF-rcut=30. However, the differences between the SREF and other ensembles for MAE and RMSE is quite small (1-2 km/hr). Lastly, we examine precipitation (Figure 19). We note, the gaps in the SRC plot are a result of SRC not able to be calculated when there are zeros in the precipitation dataset (during summer in this region). 28 Figure 19. Temporal comparison of precipitation forecasting performance of the equally weighted short-range ensemble forecast (SREF) and four weighted ensembles: Value-Based Logistic Function for k=40, k=100, and k=200(VLFk40; VLFk100; VLFk200); the Rank Based Logistic Function for rcut=30 and rcut=15 (RLFrcut30; RLFrcut15); the Sliding Window-weekly and Sliding Window-monthly (SW-Weekly; SW-Monthly); and the Inverse Error Metric (IEM). Here, missing bars for all ensembles indicated a calculation error due to zeros in the dataset; whereas, 1-2 bars missing indicated missing data from those ensembles. In almost all cases the SREF and weighted ensembles perform similarly for all forecasts during the study period. The range for MAE and RMSE is near zero to 1.75 mm/hr; and SRC ranges from -0.2 to 0.3. Precipitation has the lowest SRC compared to temperature and wind speed. Similar to wind speed, there are a few cases where the SW-weekly and RLF-rcut=30 performs the worst of any of the ensembles; however, this performance amounts to an extra error of less than 0.2 mm/hr (or one extra tipping bucket tip per hour). This is not a significant amount of error. The next section with further test this significance. 4.1.3 Forecast Verification and Statistical Significance Here a more comprehensive comparison of errors between weighted ensemble types and the SREF will be shown to determine the statistical significance of the results indicated in section 4.1.2. Figure 20 shows the comparison of all 4 weighted ensembles compared to the SREF for temperature. 29 Figure 20. Verification for temperature of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. In Figure 20, we see that all the weighted ensembles and SREF have a similar distribution of errors and correlation, including similar numbers of minimal outliers (1-4). The mean MAE and RMSE are 1°C and the mean SRC is 0.25. Table 3 summarizes the results of the T-Tests that were undertaken to determine if the weighted ensemble reduced errors significantly compared to the SREF. These results indicate that none of the weighted ensembles perform significantly better than the SREF for temperature (pvalue > 0.05). Table 3. T-Test Results for comparison of temperature verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. Value-Based Logistic Function Inverse Error Metric Rank-Based Logistic Function Sliding Window k=40 k=100 k=200 rcut=30 rcut=15 Weekly Monthly MAE 0.44/0.66 0.44/0.67 0.43/0.67 0.29/0.76 0.23/0.82 0.74/0.46 0.66/0.51 0.50/0.62 RMSE 0.49/0.63 0.48/0.63 0.48/0.63 0.34/0.74 0.29/0.77 0.79/0.43 0.63/0.53 0.54/0.59 SRC 0.19/0.85 0.19/0.85 0.19/0.85 0.08/0.94 0.15/0.88 0.25/0.81 0.33/0.97 0.03/0.98 * Pink boxes are statistically significant and blue boxes are not. Next, Figure 21 shows the comparison of all 4 weighted ensembles compared to the SREF for wind speed. 30 Figure 21. Verification for wind speed of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. In Figure 21, errors and correlation distributions are similar for all weighted ensembles and the SREF for wind speed, including similar numbers of minimal outliers (1-4). The mean MAE and RMSE are 2.5 km/hr and the mean SRC is 0.1. Table 4 summarizes the results of the T-Tests (statistic/p-value) that were undertaken to determine if the weighted ensemble reduced errors significantly compared to the SREF. These results indicate that none of the weighted ensembles perform significantly better than the SREF for wind speed (pvalue > 0.05). Table 4. T-Test Results for comparison of wind speed verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. Value-Based Logistic Function Inverse Error Metric Rank-Based Logistic Function Sliding Window k=40 k=100 k=200 rcut=30 rcut=15 Weekly Monthly MAE 0.26/0.79 0.26/0.80 0.26/0.79 0.26/0.80 0.77/0.44 0.87/0.39 0.23/0.82 0.49/0.63 RMSE 0.29/0.77 0.29/0.78 0.29/0.77 0.28/0.78 0.83/0.41 0.96/0.34 0.26/0.80 0.50/0.62 SRC 0.04/0.97 0.04/0.97 0.04/0.97 0.04/0.97 0.28/0.78 0.16/0.88 0.25/0.81 0.31/0.76 * Pink boxes are statistically significant and blue boxes are not. Lastly, Figure 22 shows the comparison of all 4 weighted ensembles compared to the SREF for precipitation. Spearman rank correlation is not included for precipitation as it is not able to be calculated when there are zeros in the dataset, which is common for the case of precipitation forecasts, especially in the dry summer months for southwestern BC. 31 Figure 22. Verification for precipitation of the equally weighted Short-Range Ensemble Forecast (SREF) all weighted ensembles: Value-Based Logistic Function (VLF) for k-parameters: 40, 100, 200; Rank-Based Logistic Function (RLF) for rcut/rhalf: 30/16, 15/7; Inverse Error Metric (IEM); and Sliding Window (SW) for weekly and monthly based windows. Above (Figure 22), we see that the error and correlation distributions are very similar for all weighted ensembles and the SREF for wind speed. Here, we do see more outliers than for the temperature and wind speed forecasts (6-12), which may be a function of the mean errors being skewed towards zero with all of the non-precipitation hours in the dataset (zeros). Because of the higher number if outliers, we see greater differences between the MAE and RMSE means. Albeit these differences are very small, 0.05 and 0.1 mm/hr, respectively. Table 5 summarizes the results of the T-Tests that were undertaken to determine if the weighted ensemble reduced errors significantly compared to the SREF. These results indicate that none of the weighted ensembles perform significantly better than the SREF for precipitation (pvalue > 0.05). Table 5. T-Test Results for comparison of precipitation verification metrics for the weighted ensembles compared to the equally weighted short-range ensemble forecast. Value-Based Logistic Function Inverse Error Metric Rank-Based Logistic Function Sliding Window k=40 k=100 k=200 rcut=30 rcut=15 Weekly Monthly MAE 0.04/0.97 0.05/0.96 0.04/0.97 0.05/0.96 0.48/0.63 0.65/0.52 0.28/0.78 0.15/0.88 RMSE 0.05/0.96 0.05/0.96 0.05/0.96 0.05/0.96 0.46/0.65 0.56/0.58 0.20/0.84 0.12/0.91 * Pink boxes are statistically significant and blue boxes are not. 4.2 Iterated-Mean Ensemble Next, the VNE and SREF are compared for temperature (Figure 23), wind speed (Figure 24) and precipitation (Figure 25). 32 Figure 23. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for temperature. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). As shown in Figure 23, it is seen that the number of ensemble members does not greatly affect the MAE or RMSE for temperature over this period, aside from a few cases. In the cases where it does, the difference in MAE and RMSE between ensembles is ~1°C. SRC has larger differences depending on the number of members though. The greatest difference in SRC between ensembles is 0.8 in December 2023, where the ensembles with the lowest number of members is greatly outperformed by those with the highest number of members. The variation in MAE, RMSE and SRC though does appear to converge on a value when the number of members increases to 30. Next, we will examine the VNE for wind speed. As shown in Figure 24, it is seen that the number of ensemble members does not greatly affect the MAE or RMSE for wind speed over this period. The differences in MAE and RMSE between the ensembles is at most 4 km/hr. SRC differences are a bit more nuanced though with the largest difference in SRC between members being 0.6. This again is between ensembles with a lower number of members (1-20) and those with the most number of members (30-50). Again, the MAE, RMSE and SRC converge on a value after the number of members exceeds 30. 33 Figure 24. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for wind speed. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). Lastly, we will examine the performance of the VNE for precipitation (Figure 25). It is shown that again the number of members does not have a large effect on the MAE or RMSE for precipitation. The difference in MAE and RMSE varies at most 0.5 mm/hr between ensembles throughout the study period. SRC varies much more though with a difference of 0.6 when comparing smaller ensembles with the larger ensembles for the end of April 2023. Again, we do see convergence in the MAE, RMSE and SRC after the number of members exceeds 30. 34 Figure 25. Comparison of varying number of members ensemble (VNE) and the short-range ensemble forecast (SREF; 51 members) for precipitation. Colors closer to red indicate a low number of members (1-10) and colors closer to purple indicate a high number of members (40-50). 35 5. Discussion 5.1 Weighting Schemes The results indicate that the weighted ensemble does not add any significance to the forecast compared to the SREF. In some cases, though, the weighted ensembles may out-perform the SREF. Examining the spatial performance of the weighted ensembles, we see that the performance is variable across the study domain, which is expected due to the complex terrain in the region and the inability of even the highest resolution models to resolve these nuances. However, we still see a common trend of the SREF outperforming the weighted ensembles for temperature and the weighted ensembles outperforming the SREF for precipitation and wind speed. However, this is not statistically significant enough (pvalue > 0.05) to warrant using the weighted ensembles over the SREF. This result is interesting as it contradicts the literature’s findings that weighted ensembles outperform the equally weighted case in most scenarios. Especially for shorter training sets. Here, we found that shorter training sets (1 week) perform similarly to longer training sets (1 season). Our result may be due to the fact the ensemble uses the “best guesses” of each agencies initial conditions, rather than perturbations of the initial conditions. This can greatly reduce error, by not allowing random noise into the members. This is evident when we look at a time series of the member’s performance over a two-year period. The performance of each member fluctuates greatly throughout the year, meaning there is no one “best member”; however, these fluctuations are not that large amongst most members. Figure 26 shows a time series of each member’s performance from 2021-2023. Figure 26. Time series of each member's performance from October 2021 to November 2023 for temperature. Examining the statistics versus time scale we can see that most of the members perform quite similarly throughout the period (October 2021-November 2023). There are some poorer performing models for some forecasts; however, they are not always performing poorly and do add value to the forecast. This 36 may explain why the weights, created with verification metrics, did not affect the ensemble as was first hypothesized. Perhaps the reason for similar performance across most of the members is a result of their construction. Most of the members are of similar model origin but rely on different initial conditions. These initial conditions often utilize the same input data, leading to limited diversity in the ensemble. To enhance the diversity in the ensemble, it might be better to instead increase the number of models run and by proxy variations in physics and parameterizations schemes. Another reason for the lack in diversity in ensemble members is due to the limited amount of data that the agencies can collect. The region to the west southwestern BC is known by local meteorologists as the ‘Pacific Data Void.’ It is a region with very limited land mass and therefore reduces the opportunity for atmospheric sounding measurements. Satellite data can collect upper atmosphere measurements and buoys can collect near surface measurements. However, there is a very large gap in the middle where there is a paucity of thermodynamic and dynamic information for data assimilation (Stull et al., 2004; Zheng et al., 2020). This Pacific data void has been long discussed in the literature and lots of work is being done to minimize this problem (Zheng et al., 2020). However, due to the large region that the data void exists in, much more work still needs to be done to reduce its impact on numerical weather prediction for western North America. 5.2 VNE When examining the VNE, we see that the number of ensemble members does have a small impact on the MAE and RMSE throughout the period and a large impact on the SRC. This is the case for all three variables. There is a convergence of MAE, RMSE and SRC though once the ensemble gets sufficiently large: the number of members exceeds 30. Which could provide a useful value moving forward in operations. It is very computationally expensive to run a 51-member ensemble. Reducing this ensemble size by 21 members, where the ensemble skill plateaus, can greatly reduce this computational load. However, this is the result of 1 year of data and should be tested on longer datasets to ensure that it is true for more than one weather regime (i.e. La Nina, EL Nino). 5.3 Weighted Ensemble Limitations There are several limitations of weighted ensembles that need to be addressed. Firstly, when creating weights in hindsight to be applied to forecasts in the future, an assumption must be made that all members will be present for all future forecasts. This is because normalized weights are made with the intention of summing all members. However, if one member is missing during this summation, the forecast will be underrepresented. In operational forecasting, it is quite common for members to not be present throughout the year due to maintenance or computer issues. This makes relying on a weighted ensemble tricky as one cannot plan for a member to be out of service. However, this can be combatted by normalizing the weights just before the weights are applied to the forecasting members for each forecast period. This of course will add additional steps but should be negligible in terms of computational time. Another limitation of weighted ensembles are the time it takes for construction. If weights are to be constructed in a sliding window approach, rather than based on past data, the forecasting time can almost double, resulting in the forecast being produced after the time horizon of interest is made. This is obviously not what a forecaster would like. 37 5.3 Study Limitations Given, the short range in data used to test the weighted ensemble, the number of conclusions that can be made from this study are limited. Due to data constraints, this study was only able to utilize one year if data for calculating the weights. This is not representative of the climatological regime and could be a reason for poor performance of the weighting ensembles. With a longer dataset, the weighted ensemble might produce more pronounced effects, especially when looking at how each model resolves larger circulation patterns and oscillations such as ENSO. Future studies should examine longer time periods to really delve into how well smaller versus larger ensembles will perform, as well as weighted versus unweighted ensembles. 38 6. Conclusion In this study weighted ensembles were explored with the aim to enhance the forecasting power of the WFRT SREF. Three different weighting seasonal schemes were investigated: a value based logistic function, a rank based logistic function and an inverse error function; and two different sliding window approaches were taken: weekly and monthly. It was determined that all four weighting schemes did not statistically significantly add value compared to the equally averaged ensemble currently used. Varying the number of ensemble members was also explored. It was found that the number of ensemble members does not greatly alter the skill of the ensemble, but the pattern of this was hard to capture given the short dataset. These results may also suggest a lack in diversity in the WFRT SREF members, as many performed quite similarly throughout the study period. However, it was noted that the error and correlation converged when the number of ensembles members was greater than 30. This could be useful in future operations if the WRFT were to reduce the number of SREF members in order to save on computational costs. Although we have drawn conclusions here, this study was able to use only two years of data (one for training and one for testing). This amount of data is unable to capture nuances in the climatology of the region, nor general circulation patterns that affect the weather on scales of greater than a year. It is recommended that a study of similar style be done including these climatological effects to be able to draw broader and more conclusive conclusions. 39 References Amarasinghe, P. A. G. M., Abeygunawardana, T. N., Jayasekara, T. N., Edirisinghe, E. A. J. P., and S. K. Abeygunawardane, 2020: Ensemble models for solar power forecasting – a weather classification approach. AIMS Energy, 8, 2, 252-271, https://doi.org/10.3934/energy.2020.2.252 Berrocal, V. J., Raftery, A. E. and T. Gneiting, 2007: Combining Spatial Statistical and Ensemble Information in Probabilistic Weather Forecasts. Mon. Wea. Rev., 135, 4, 1386-1402, https://doi.org/10.1175/MWR3341.1 Bishop, C. H., and K. T. Shanley 2008: Bayesian Model Averaging’s Problematic Treatment of Extreme Weather and a Paradigm Shift That Fixes It. Mon. Wea. Rev., 136, 12, 4641-4652, https://doi.org/10.1175/2008MWR2565.1 Bradley, A. A., Habib, M., and S. S. Schwartz, 2015: Climate index weighting of ensemble streamflow forecasts using a simple Bayesian approach. Water Resources Research, 51, 9, 7382-7400, https://doi.org/10.1002/2014WR016811 Brooks, H. E. and C. A. Doswell III, 1996: A Comparison of Measures-Oriented and Distributions-Oriented Approaches to Forecast Verification. Weather and Forecasting, 11, 3, 288-303, https://doi.org/10.1175/1520-0434(1996)011<0288:ACOMOA>2.0.CO;2 R. Buizza, 2008: Comparison of a 51-Member Low-Resolution (TL399L62) Ensemble with a 6-Member High-Resolution (TL799L91) Lagged-Forecast Ensemble. Mon. Wea. Rev., 136, 9, 3343-3362, https://doi.org/10.1175/2008MWR2430.1 Canada, 2013: Flooding events in Canada: prairie provinces. Accessed 18 November 2023, https://www.canada.ca/en/environment-climate-change/services/water-overview/quantity/floods/events-prairie-provinces.html Canada, 2021: Canada’s top 10 weather stories of 2021. Accessed 18 November 2023, https://www.canada.ca/en/environment-climate-change/services/top-ten-weather-stories/2021.html#toc2 Caron, J.-F., Zadra, A., Anselmo, D., Milewski, T., and A. Patoine, 2016: Regional Deterministic Prediction System (RDPS) – Update from version 4.2.0 to version 5.0.0, Technical note, Canadian Meteorological Center, Environment and Climate Change Canada. Accessed 18 November 2023, https://collaboration.cmc.ec.gc.ca/cmc/CMOI/product_guide/docs/lib/technote_rdps-500_20160907_e.pdf Côté, J., S. Gravel, A. Méthot, A. Patoine, M. Roch, and A. Staniforth, 1998: The operational CMC-MRB Global Environmental Multiscale (GEM) model: Part I - Design considerations and formulation, Mon. Wea. Rev. 126, 1373-1395, https://doi.org/10.1175/1520-0493(1998)126<1373:TOCMGE>2.0.CO;2 Déqué, M., Dreveton, C., Braun, A., and D. Cariolle, 1994: The ARPEGE/IFS atmosphere model: a contribution to the French community climate model. Climate Dynamics, 10, 249-266, https://doi.org/10.1007/BF00208992 Doycheva, K., Horn, G., Koch, C., Chumann, A. and M. Konig, 2017: Assessment and weighting of meteorological ensemble forecast members based on supervised machine learning with application to 40 runoff simulations and flood warning. Advanced Engineering Informatics, 22, 427-439, https://doi.org/10.1016/j.aei.2016.11.001 Du, J., and B. Zhou, 2011: A Dynamical Performance-Ranking Method for Predicting Individual Ensemble Member Performance and Its Application to Ensemble Averaging. Mon. Wea. Rev., 139, 10, 3284-3303, https://doi.org/10.1175/MWR-D-10-05007.1 ECCC, 2017: Top weather events of the 20th century. Accessed 18 November 2023, http://wayback.archive-it.org/7084/20170925152846/https://www.ec.gc.ca/meteo-weather/default.asp?lang=En&n=6A4A3AC5-1#tab5 Gebhardt, C., Theis, S. E., Paulat, M. and Z. B. Bouallegue, 2011: Atmospheric Research, 100, 2-3, 168-177, https://doi.org/10.1016/j.atmosres.2010.12.008 Gensler, A., and B. Sick, 2016: Forecasting Wind Power – An Ensemble Technique with Gradual Cooperative Weighting Based on Weather Situation [Paper presentation]. 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada, https://doi.org/10.1109/IJCNN.2016.7727855 Gneiting, T., Raftery, A. E., Westveld III, A. H. and T. Goldman, 2005: Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation. Mon. Wea. Rev., 133, 5, 1098-1118, https://doi.org/10.1175/MWR2904.1 Gneiting, T., and A. E. Raftery, 2005: Weather Forecasting with Ensemble Methods. Science, 310, 5746, 248-249, https://doi.org/10.1126/science.1115255 Grell, G. A., Dudhia, J., and D. Stauffer, 1994: A description of the fifth-generation Penn State/NCAR Mesoscale Model (MM5) (No. NCAR/TN-398+STR). University Corporation for Atmospheric Research, https://doi.org/10.5065/D60Z716B Greybush, S. J., Haupt, S. E. and G. S. Young, 2008: The Regime Dependence of Optimally Weighted Ensemble Model Consensus Forecasts of Surface Temperature. Weather and Forecasting, 23, 6, 1146-1161, https://doi.org/10.1175/2008WAF2007078.1 Guerra, J. A., Murray, S. A., Bloomfield, D. S., and P. T. Gallagher, 2020: Ensemble forecasting of major solar flares: methods for combining models. J. Space Weather Space Clim, 10, 38, https://doi.org/10.1051/swsc/2020042 Guerra, J. A., Pulkkinen, A. and V. M. Uritsky, 2015: Ensemble forecasting of major solar flares: First Results. Space Weather, 13, 626-642, https://doi.org/10.1002/2015SW001195 Hogan, R. J., Ferro, C. A. T., Jolliffe, I. T. and D. B. Stephenson, 2010: Equitability Revisited: Why the “Equitable Threat Score” Is Not Equitable. Weather and Forecasting, 25, 2, 710-726, https://doi.org/10.1175/2009WAF2222350.1 Hogan, T. F., Liu, M., Ridout, J. A., Peng, M. S., Whitcomb, T. R., Ruston, B. C., Reynolds, C. A., Eckermann, S. D., Moskaitis, J. R., Baker, N. L., Mccormack, J. P., Viner, K. C., Mclay, J. G., Flatau, M. K., E. Kalnay, 2019: Historical perspective: earlier ensembles and forecasting forecast skill. Q. J. R. Meteorol Soc, 145, Suppl. 1, 25-34, https://doi.org/10.1002/qj.3595 41 Jacobs, Q., 2015: Evaluating Human-Processed Forecasts Provided to the Aviation Industry in South Africa [Masters thesis, University of Pretoria]. https://repository.up.ac.za/bitstream/handle/2263/45916/Jacobs_Evaluating_2015.pdf?sequence=1&isAllowed=y Kang, S. -D., Shin, D. W., Cocke, S., Kim, H. -D. and W. -S. Jung, 2011: Comparison of ensemble methods for summertime numerical weather prediction over East Asia. Meteorology and Atmospheric Physics, 113, 27-38, https://doi.org/10.1007/s00703-011-0148-6 Kelsch, M. et al., 2021: Chapter 4. Ensemble Dispersion and Ensemble Spread [MOOC]. In Foundations in Probabilistic Forecasting. MetEd, Accessed 13 November 2023, https://www.meted.ucar.edu/nwp/foundations_probabilistic/vignettes/navmenu.php?tab=4 Krishnamurti, T. N. et al., 2003: Improved Skill for the Anomaly Correlation of Geopotential Heights at 500 hPa. Mon. Wea. Rev., 131, 6, 1082-1102, https://doi.org/10.1175/1520-0493(2003)131<1082:ISFTAC>2.0.CO;2 Lee, J. A., Kolczynski, W. C., McCandless, T. C., and S. E. Haupt, 2012: An Objective Methodology for Configuring and Down-Selecting an NWP Ensemble for Low-Level Wind Prediction. Mon. Wea. Rev., 140, 7, 2270-2286, https://doi.org/10.1175/MWR-D-11-00065.1 Liu, L., Gao, C., Xuan, W., and Y. -P. Xu, 2017: Evaluation of medium-range ensemble flood forecasting based on calibration strategies and ensemble methods in Lanjiang Basin, Southeast China. Journal of Hydrology, 554, 233-250, https://doi.org/10.1016/j.jhydrol.2017.08.032 E. N. Lorenz, 1965: A study of the predictability of a 28-variable atmospheric model. Tellus, 17, 3, 321-333, https://doi.org/10.1111/j.2153-3490.1965.tb01424.x Lu, C., Yuan, H., Schwartz, B. E., and S. G. Benjamin, 2007: Short-Range Numerical Weather Prediction Using Time-Lagged Ensembles. Weather and Forecasting, 22, 3, 580-595, https://doi.org/10.1175/WAF999.1 Milinski, S., Maher, N., and D. Olonscheck, 2020: How large does a large ensemble need to be? Earth Syst. Dynam., 11, 885–901, https://doi.org/10.5194/esd-11-885-2020 Murphy, A. H., 1988: Skill Scores Based on the Mean Square Error and Their Relationships to the Correlation Coefficient. Mon. Wea. Rev., 116, 12, 2417-2424, https://doi.org/10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2 Najafi, M. R., Moradkhani, H. and T. C. Piechota, 2012: Ensemble Streamflow Prediction: Climate signal weighting methods vs. Climate Forecast System Reanalysis. Journal of Hydrology, 442-443, 105-116, https://doi.org/10.1016/j.jhydrol.2012.04.003 NCEP, 2023a: The Global Forecast System (GFS) Documentation. National Centers for Environmental Prediction, Accessed 18 November 2023, https://www.emc.ncep.noaa.gov/emc/pages/numerical_forecast_systems/gfs/documentation.php NCEP, 2023b: North American Mesoscale (NAM) Documentation. National Centers for Environmental Prediction. Accessed 18 November 2023, https://www.emc.ncep.noaa.gov/emc/pages/numerical_forecast_systems/nam.php 42 Pagowski, M. et al., 2005: A simple method to improve ensemble-based ozone forecasts. Geophysical Research Letters, 32, 7, https://doi.org/10.1029/2004GL022305 Pasaric, Z. and J. Juras, 2011: Comments on “Applying a General Analytic Method for Assessing Bias Sensitivity to Bias-Adjusted Threat and Equitable Threat Scores.” Weather and Forecasting, 26, 1, 122-125, https://doi.org/10.1175/2010WAF2222453.1 Patra, A. K. et al., 2013: Challenges in Developing DDDAS based Methodology for Volcanic Ash Hazard Analysis – Effect of Numerical Weather Prediction Variability and Parameter Estimation. Procedia Computer Science, 18, 1871-1880, https://doi.org/10.1016/j.procs.2013.05.356 Raferty, A. E., Gneiting, T., Balabdaoui, F. and M. Polakowski, 2005: Using Bayesian Model Averaging to Calibrate Forecast Ensembles. Mon. Wea. Rev., 133, 5, 1155-1174, https://doi.org/10.1175/MWR2906.1 Robertson, A. W., Lall, U., Zebiak, S. E. and L. Goddard, 2004: Improved Combination of Multiple Atmospheric GCM Ensembles for Seasonal Prediction. Mon. Wea. Rev., 132, 12, 2732-2744, https://doi.org/10.1175/MWR2818.1 Salehi, M., Leckie, C.A., Moshtaghi, M., and T. Vaithianathan, 2014: A Relevance Weighted Ensemble Model for Anomaly Detection in Switching Data Streams. In: Tseng, V.S., Ho, T.B., Zhou, ZH., Chen, A.L.P., Kao, HY. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2014. Lecture Notes in Computer Science(), vol 8444. Springer, Cham, https://doi.org/10.1007/978-3-319-06605-9_38 J. Schneider, 1997: Cross Validation [Online Tutorial]. In Locally Weighted Learning, Accessed 18 November 2023, https://www.cs.cmu.edu/~schneide/tut5/node42.html#SECTION00092000000000000000 Schober, P., Boer, C., and L. A. Schwarte, 2018: Correlation Coefficients: Appropriate Use and Interpretation. Anesthesia & Analgesia, 126, 5, 1763-1768, https://doi.org/10.1213/ANE.0000000000002864 Skamarock, W. C., J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, M. G Duda, X.-Y. Huang, W. Wang, and J. G. Powers, 2008: A Description of the Advanced Research WRF Version 3. NCAR Tech. Note NCAR/TN-475+STR, 113, https://doi.org/10.5065/D68S4MVH Skamarock, W. C., Klemp, J. B., Duda, M. G., Fowler, L. D., Park, S. -H., and T. D. Ringler, 2012: A Multiscale Nonhydrostatic Atmospheric Model Using Centroidal Voronoi Tesselations and C-Grid Staggering. Mon. Wea. Rev., 140, 9, 3090–3105, https://doi.org/10.1175/MWR-D-11-00215.1 Skamarock, W. C., Klemp J. B., Dudhia J., Gill D. O., Liu Z., Berner J., Wang W., Powers J. G., Duda M. G., Barker D. M., and X.-Y. Huang, 2019: A Description of the Advanced Research WRF Version 4. NCAR Tech. Note NCAR/TN-556+STR, 145, https://doi.org/10.5065/1dfh-6p97 Sofiati, I. and A. Nurlatifah, 2019: The prediction of rainfall events using WRF (weather research and forecasting) model with ensemble technique [Paper presentation]. IOP Conf. Series: Earth and Environmental Science, https://doi.org/10.1088/1755-1315/374/1/012036 Stanger, J., Finney, I., Weisheimer, A. and T. Palmer, 2019: Optimizing the use of ensemble information in numerical weather forecasts of wind power generation. Environmental Research Letters, 14, 12, https://doi.org/10.1088/1748-9326/ab5e54 43 Stensrud, D. J., Bao J, -W. and T. T. Warner, 2000: Using Initial Condition Model Physics Perturbations in Short-Range Ensemble Simulations of Mesoscale Convective Systems. Mon. Wea. Rev., 128, 7, https://doi.org/10.1175/1520-0493(2000)128<2077:UICAMP>2.0.CO;2 Stensrud, D. J. and M. S. Wandishin, 2000: The Correspondence Ratio in Forecast Evaluation. Weather and Forecasting, 15, 5, 593-602, https://doi.org/10.1175/1520-0434(2000)015<0593:TCRIFE>2.0.CO;2 Stephenson, D. B., 2000: Use of the “Odds Ratio” for Diagnosing Forecast Skill. Weather and Forecasting, 15, 2, 221-232, https://doi.org/10.1175/1520-0434(2000)015<0221:UOTORF>2.0.CO;2 Suh, M. -S. et al., 2012: Development of New Ensemble Methods Based on the Performance Skills of Regional Climate Models Over South Korea. Journal of Climate, 25, 20, 7067-7082, https://doi.org/10.1175/JCLI-D-11-00457.1 Thorey, J., Chaussin, C. and V. Mallet, 2018: Ensemble forecast of photovoltaic power with online CRPS learning. International Journal of Forecasting, 34, 4, 762-773, https://doi.org/10.1016/j.ijforecast.2018.05.007 Weyn, J. A., Durran, D. R., Caruana, R. and N. Cresswel-Clay, 2021: Sub-Seasonal Forecasting with a Large Ensemble of Deep-Learning Weather Prediction Models. Journal of Advances in Modeling Earth Systems, 13, e2021MS002502, https://doi.org/10.1029/2021MS002502 Whitaker, J. S., and A. F. Loughe, 1998: The Relationship between Ensemble Spread and Ensemble Mean Skill. Mon. Wea. Rev., 126, 12, 3292-3302, https://doi.org/10.1175/1520-0493(1998)126<3292:TRBESA>2.0.CO;2 WWRP JWGFVR, World Weather Research Program Joint Working Group on Forecast Verification Research, 2015: WWRP/WGNE Joint Working Group on Forecast Verification Research. https://www.cawcr.gov.au/projects/verification/#:~:text=What%20is%20forecast%20verification%3F,the%20quality%20of%20a%20forecast Xu, l., Chen, C., and S. W. Chang, 2014: The Navy Global Environmental Model. Oceanography, 27, 3, 116–125, http://www.jstor.org/stable/24862194 Zängl, G., Reinert, D., Ripodas, P., and M. Baldauf, M, 2015: The ICON (ICOsahedral Non-hydrostatic) modelling framework of DWD and MPI-M: Description of the non-hydrostatic dynamical core. Quarterly Journal of the Royal Meteorological Society, 141, 687, 563–579, https://doi.org/10.1002/qj.2378 Zhang, J., Chen, J., Li. X., and H. Chen, 2020: Combining Postprocessed Ensemble Weather Forecasts and Multiple Hydrological Models for Ensemble Streamflow Predictions. Journal of Hydrological Engineering, 25, 1, https://doi.org/10.1061/(ASCE)HE.1943-5584.0001871 Zheng, M., Monache, L. D., Ralph, F. M., Cornuelle, B., Tallapragada, V. Haase, J. S., Wilson, A. M., Mazloff, M., Subramanian, A. C., and F. Cannon, 2020: Data Gaps withing Atmospheric Rivers over the Northeastern Pacific. Bulletin of the American Meteorological Society, 103, 3, https://doi.org/10.1175/BAMS-D-19-0287.1 44 Appendix A. Station Observation Metadata 45 46 Appendix B. Verification Metric Equations For continuous verification metrics, 𝐹𝑖 is the forecast for location or time 𝑖, 𝑂𝑖 is the observations for a location or time 𝑖, and 𝑁 is the total number of forecast and observation pairs being verified. Mean Absolute Error (MAE)1: 𝑀𝐴𝐸 =1𝑁 ∑|𝐹𝑖 − 𝑂𝑖|𝑁𝑖=1 Root Mean Squared Error (RMSE)1: 𝑅𝑀𝑆𝐸 = √1𝑁 ∑(𝐹𝑖 − 𝑂𝑖)2𝑁𝑖=1 Spearman Rank Correlation (SRC)2: 𝑆𝑅𝐶 = 1 − 6 ∑ (𝐹𝑅𝑎𝑛𝑘 − 𝑂𝑅𝑎𝑛𝑘)𝑖2𝑁𝑖=1𝑁(𝑁2 − 1) Where 𝐹𝑅𝑎𝑛𝑘 is the rank of each forecast and 𝑂𝑅𝑎𝑛𝑘) is the rank of each observation. Categorical scores are based on values found in contingency tables1. An example contingency table is shown: Observation Yes No Forecast Yes Hit False Alarm No Miss Correct Negatives Here we use 3 category contingency tables for precipitation and wind speed separately as follows: Thresholds for these categorical scores were based on following two schemes: Precipitation Observation No Rain Light Rain Heavy Rain Forecast No Rain Correct Negative Miss Miss Light Rain False Alarm Hit Miss Heavy Rain False Alarm False Alarm Hit Threshold Definitions: • No rain. • Light rain: precipitation value falls in the bottom 2/3 of the probability distribution. • Heavy rain: precipitation value falls in the top 1/3 of the probability distribution. 47 Wind Speed Observation Calm Wind Light Wind Strong Wind Forecast Calm Wind Correct Negative Miss Miss Light Wind False Alarm Hit Miss Strong Wind False Alarm False Alarm Hit Threshold Definitions (based on Beaufort wind scale): • Calm wind: winds speeds are less than or equal to 5 km/hr. • Light wind: wind speeds are between 5 km/hr and 40 km/hr. • Strong wind: winds speeds are greater than 40 m/hr. These table values are then used in the scores as follows: Probability of Detection (POD) 1: 𝑃𝑂𝐷 = ℎ𝑖𝑡𝑠ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠 Eq. B1 Probability of False Detection (POFD) 1: 𝑃𝑂𝐹𝐷 = 𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠 Eq. B2 Pierce Skill Score (PSS) 1: 𝑃𝑆𝑆 = ℎ𝑖𝑡𝑠ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠− 𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠 Eq. B3 Gilbert Skill Score (GSS) 1: ℎ𝑖𝑡𝑠𝑟𝑎𝑛𝑑𝑜𝑚 = (ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠)+(ℎ𝑖𝑡𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠)ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠+𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠 Eq. B4a 𝐺𝑆𝑆 = ℎ𝑖𝑡𝑠− ℎ𝑖𝑡𝑠𝑟𝑎𝑛𝑑𝑜𝑚ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠− ℎ𝑖𝑡𝑠𝑟𝑎𝑛𝑑𝑜𝑚 Eq. B4b Critical Success Index (CSI) 1: 𝐶𝑆𝐼 = ℎ𝑖𝑡𝑠ℎ𝑖𝑡𝑠+𝑚𝑖𝑠𝑠𝑒𝑠+𝑓𝑎𝑙𝑠𝑒 𝑎𝑙𝑎𝑟𝑚𝑠 Eq. B5 1. WWRP JWGFVR, World Weather Research Program Joint Working Group on Forecast Verification Research, 2015: WWRP/WGNE Joint Working Group on Forecast Verification Research. https://www.cawcr.gov.au/projects/verification/#:~:text=What%20is%20forecast%20verification%3F,the%20quality%20of%20a%20forecast 2. McAleer, P., 2023: A Handy Workbook for Research Methods & Statistics. https://psyteachr.github.io/handyworkbook/spearman-correlation.html
Thesis/Dissertation
eng
Atmospheric Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Stull, Roland B., 1950-
Investigating the utility of weighting schemes on a 51-member short-range ensemble forecast
Text