In the last years, several events of heavy precipitation over the Basque Country have caused different flood episodes. The complex orography and rivers characteristics, among other factors, favour the occurrence of these episodes. Figure 1 shows the location of the studied river basins. In order to understand the occurrence and dangerousness of these flood episodes, a full study was made including synoptic, mesoscale information and local meteorological characteristics.

In the Basque Country Agency (Euskalmet), different Numerical Weather Prediction (NWP) models run operationally (Egaña et al., 2008; Gaztelumendi et al., 2007, 2009; Gelpi et al., 2007, 2013). One of them is the Weather Research and Forecasting model (WRF) (Skamarock et al., 2005). In this paper, we present a preliminary comparison of different microphysics and cumulus parameterization schemes in the WRF quantity precipitation forecast (QPF) for 21st century extreme precipitation events in the Basque Country. We focus on particular river basins in the Basque Country for a selection of severe episodes. Analysis and validation are based mainly on rain data from the Basque Country Automatic Weather Station (AWS) Mesonetwork (Gaztelumendi et al., 2003).

The purpose of this study is to know skill and reliability of different WRF parameterization configurations on extreme precipitation forecast, allowing us to improve the forecast tasks.

The rest of the paper is organised as follows. Section 2 describes the experimental framework used to evaluate performance of parameterization configurations. Section 3 describes the validation process and observational data. Section 4 describes the results. Finally, Sect. 5 presents some conclusions and recommendations for future work.

The performance of several cumulus and microphysics parameterizations is studied for extreme rainfall events. Combinations among two Cumulus Parameterization (CP) and five Microphysics Parameterization (MP) schemes result in ten different physics configurations for WRF-system.

Planetary boundary layer YSU-PBL, RRTM/Dudhia radiation scheme and Noah land surface model parameterizations remain unaltered for the whole set of experiments. These parameterizations are used in WRF-Euskalmet since its operational installation.

The main characteristics of WRF-Euskalmet, based on WRF-ARW 3.2.1, are Lambert projection and “two-way” nesting technique with a 3 ratio for the four nested domains. Grid resolutions are 81 (55 × 55), 27 (55 × 55), 9 (55 × 55) and 3 km (58 × 58), respectively.

Combinations used on experiments (E1–E10), showed in Table 1, were chosen based on literature review (Cossu and Hocke, 2014; Gallus Jr. and Pfeifer, 2008; Gilliland and Rowe, 2007; Hong et al., 2010; Jankov et al., 2005; Otkin et al., 2006). E0 experiment corresponds to WRF-Euskalmet operational configuration. The cumulus parameterization scheme used is a combination of Kain–Fritsch for the coarser domains (81 and 27 km) and Grell–Deyeni for the finer domains (9 and 3 km), with WSM3 scheme for microphysics parameterization.

The initial and boundary conditions for the coarser grid are obtained from the Global Forecast System (GFS), run by NCEP (National Center for Environmental Prediction), 1-degree analysis data.

MP schemes selected are: WRF Single-Moment 3-class (WSM3), Purdue Lin (LIN), WRF Single-Moment 6-class (WSM6), New Thompson (NT) and WRF Double-Moment 6-class scheme (WDM6).

The WSM3 (Hong et al., 2004) categories are vapor, cloud water/ice, and rain/snow. The cloud ice and cloud water are counted as the same category, and they are distinguished by temperature. The WSM6 (Hong and Lim, 2006), LIN (Chen and Sun, 2002), and NT (Thompson et al., 2008) schemes contain prognostic equations for cloud water, rain water, ice, snow, and graupel mixing ratios. NT scheme also predicts the total concentration of ice. The WDM6 scheme (Lim and Hong, 2010) is the extended version of the WSM6 adding the prognostic of cloud and rainwater together with the cloud condensation nuclei (CCN) concentration. The inclusion of prognostic equations for the total concentration of each species is computationally demanding but it allows for a more realistic treatment of many microphysical processes.

CP schemes are: Kain–Fritsch (KF), Betts–Miller–Janjic (BMJ) and Grell 3-D schemes.

KF scheme is a shallow sub-grid scheme that uses a mass flux approach with downdrafts and CAPE removal timescale closure, includes condensed and gaseous water detrainment, and the clouds persist over the convective time scale (Kain, 2004; Kain and Fritsch, 1990). BMJ scheme is an adjustment type scheme that generates deep and shallow convection. Relaxing is applied towards variable temperature and humidity profiles determined from thermodynamic considerations (Janjic, 1994). GD, Grell 3-D is an improved version of the GD (Grell–Devenyi) scheme.

A set of 35 severe weather episodes of heavy/persistent precipitation in the Basque Country for the 21st century have been selected. The selection criterion used is related to precipitation episodes that have caused flooding and/or damages in river basins that include highly populated areas, as Bilbao and San Sebastian surroundings. An episode of precipitation may correspond to one or more days (Table 2).

For validation purposes, objective point to point comparisons are made between simulated daily precipitation amount values versus observed at some selected stations from the AWS Basque Mesonetwork (Gaztelumendi et al., 2003). The stations were selected to ensure that the observations are representative for Kadagua and Urumea river basins (see Fig. 1).

To carry out the validation and analysis process, several scatter plots and graphs are prepared, including statistical continuous parameters, to know the behaviour in quantity precipitation forecast as MAE (Mean Absolut Error) of the absolute values of the individual forecast errors. RMSE (Root Mean Square Error) is more sensitive to large forecast errors than MAE. NRMSE (Normalized RMSE) facilitates the comparison between datasets or models with different scales; the approach taken is to normalise by the mean value of the observations.

More interesting to us than errors in quantity, is proper detection of severe weather forecast events operationally. A useful summary of the forecast of observed weather events can be presented in a contingency table, which does not constitute a verification method by itself, but provides the basis from which useful scores can be obtained.

Contingency tables are useful to understand dichotomous (yes/no) forecasts, yes (event will happen), no (event will not happen), rain is a common example of this type. The four combinations of forecasts (yes or no) and observations (yes or no) are: hits (event was forecast to occur, and did occur), false alarms (event was forecast to occur, but did not occur), misses (event was forecast not to occur, but did occur) correct non-events (event was forecast not to occur, and did not occur).

A large variety of categorical statistics are computed from the elements in the contingency tables to describe particular aspects of forecast performance. We have worked with Proportion Correct score (PC), Probability of Detection (POD), False Alarm Rate (FAR), Critical Success Index (CSI) and Heidke Skill Score (HSS). Proportion Correct score indicates what fraction of the forecasts was correct. It is simple and intuitive, and heavily influenced by the most common category (possible values Perfect = 0, No skill = 1). Probability of Detection indicates what fraction of the observed yes events was correctly forecast (Perfect = 1, No skill = 0). POD is sensitive to hits but ignore false alarms. It is good for rare events and should be used in conjunction with the False Alarm Ratio index. False Alarm Ratio indicates what fraction of the predicted yes events actually did not occur, i.e. the fraction of non-events which were forecast as false alarms. It is sensitive to false alarms but ignore miss values (Perfect = 0, No skill = 1). Critical Success Index indicates how well the forecast yes events corresponded to the observed yes events. It measures the fraction of observed and forecast events that were correctly predicted. It is quite sensitive to hits and penalizes both misses and false alarm (Perfect = 1, No skill = 0). Heidke Skill Score indicates the accuracy of the forecast in predicting the correct category, relative to that of random choice. It measures the fraction of correct forecasts after removing those forecasts that would be correct due to purely random chance. (Perfect = 1, No skill = 0).

The Basque Meteorology Agency (Euskalmet) is responsible for issuing severe weather warnings in the Basque Country area. To assess the ability of the different configurations in defining the level of risk of adverse events rainfall, 4-category contingency tables were created, based on Euskalmet warning system thresholds (Gaztelumendi et al., 2012) and also 11-category with regular 20 mm intervals, summarized in Table 3. PC, FAR, POD, HSS and CSI indexes related to contingency tables were calculated (see Figs. 4–6).

The analysis of the results show that the 9 km resolution domain, for most of the events and experiments, works better for QPF (Quantitative Precipitation Forecast) than other domains, including 3 km resolution. This behaviour can be explained by the validation methodology, based on rain-gauge comparison that penalizes high resolution precipitation patterns with good shape but with poor accuracy, and also by the characteristics of microphysics and cumulus parameterization for resolutions smaller than 10 km.

Scatter-plots give information about the correspondence between forecasts and observations and offer the advantage of presenting in a synthetic way all the statistical information in the data set. An accurate forecast will have points on or near the diagonal.

Figure 2 shows the scatter plots for the whole set of experiments (E0–E10), using all the events and stations of both river basins. If we focus on highest values of the y axis (observed), we will have a qualitative verification of configuration performance on extreme precipitation forecast. Statistical error values are summarized in Table 4.

Most of the BMJ cumulus parameterizations (E1–E5) produce underestimation of precipitation for the majority of the events in the eastern river basin (Urumea), especially using NTH (E4) and WDM6 (E5) microphysics parameterization schemes. The exception is the WSM3 + BMJ (E1). The KF cumulus parameterizations (E0, E6–E10) perform forecasts more adequately. The configuration E9 with NTH microphysics parameterization makes the best characterization for 300–400 mm episodes. These episodes of highest precipitation correspond to the eastern river basin (Urumea). WSM3 (E6), LIN (E7) and WDM6 (E10) generate very large overestimation for a single episode.

In the western river basin (Kadagua), for events exceeding the 100 mm (not observed events exceeding 200 mm), the model configurations with BMJ cumulus parameterization underestimate precipitation forecast, more noticeable with WSM3 (E1), LIN (E2) and NTH (E4) microphysics schemes. Similar behaviour is observed in the NTH + KF (E9) configuration. The episodes with greater amount of precipitation are properly simulated by the reference configuration WSM3 + KF - Grell (E0).

The configurations with WSM3 microphysics scheme (E0, E1 and E6) seem to works properly, regardless of the cumulus parameterization used (see Figs. 2–5 and Table 4), in both river basins. The NRMSE values in the western river basin are lower for WSM3 + KF (E6), and WSM6 + KF (E8) configurations. In the eastern river basin the configurations with lower NRMSE are LIN + BMJ (E2) and WSM3 + KF - GD (E0), see Fig. 3.

Analysing indexes coming from contingency table, Proportion Correct (PC) index values are higher in the western river basin than the eastern. All configurations show similar values for each of the river basins. Most of the HSS indexes are higher for KF cumulus parameterization configurations and higher for the western river basin than for the eastern one (see Fig. 4).

For 11-category daily precipitation contingency tables (see Fig. 6), increasing the event severity causes skill indexes to worsen. Probability of detection (POD) of yellow-orange-red cases is higher in KF cumulus parameterization (E0, E6–E10) than in BMJ cumulus parameterization configurations (E1–E5), pointing out that the precipitation events in the 0–20 mm day-1 interval are better predicted by BMJ configurations. LIN + BMJ (E2), WSM3 + KF (E6) and NTH + KF (E9) configurations show no skill in red events detection, while KF - GD + WSM3 (E0) and KF + LIN (E7) configurations seem to be the best configurations with a general behaviour from green to red but with poor results in the eastern river basin (Urumea) with higher NRMSE values.

A preliminary subjective validation was made by comparison of simulated precipitation patterns against observed precipitation fields. Observed maps have been generated using geostatistical techniques (Hernandez et al., 2003). In Fig. 7, some examples of forecasted precipitation patterns vs. observed are showed. Maximum values in eastern Basque Country are correctly located but underestimated, while quantity differences between north and south precipitation patterns (top panels) are properly simulated. Maximums of rainfalls are acceptably simulated in location and quantity, as well as other secondary patterns (medium panels). For maximums located in the east and centre of the Basque Country, the amount of precipitations and its location is correctly forecasted (bottom panels).

Is this work, the performance of different configurations in predicting adverse precipitation episodes on daily operational numerical weather prediction has been studied.

Configurations with KF scheme for cumulus parameterization forecast better the quantity of precipitation than BMJ scheme configurations. The configurations present better performance in the western river basin than in the eastern one. Depending on the severity level and the river basin analysed various microphysical schemes show the best behaviour, implying that a single configuration is not accurate enough to simulate all analysed events. For extreme precipitation events, neither acceptable nor overall results were found.

With these results an optimal setup is not possible for operational usage in the Basque country, although poor results of BMJ cumulus parameterization configurations advise against its use.

We have just begun to analyse simulation results and to extract some preliminary conclusions. To obtain full concussions for parameterizations performance in operational models, further research is planned including different aspects as commented in following paragraph,.

The number of events and the stratification should be increased, including more severe events and other categorization groups (synoptic forcing characteristics, seasonal, weather types, etc.). The subsets should contain enough cases to produce reliable verification results. If not possible, as usual for rare events, we need to include quantitative uncertainty estimations of the verification results. This will allow us to judge whether it is likely that differences in model performance are real or just an artificial outcome of sampling variability.

The number of experiments should be increased, to also test the influence of different planetary boundary layer schemes in the precipitation forecast of extreme events around the selected areas.

Some skill scores related to persistence should be used to put verification results in perspective and to show the usefulness of the analysed options for operational purposes

NWP data used in this study are not publicly available, but they are archived in the Basque Meteorology Agency (Euskalmet). Observed data is available at http://www.euskalmet.euskadi.net/s07-5853x/es/meteorologia/lectur.apl?e=5.