Observation Preprocessing System for RC LACE (OPLACE)

Meteorological observations are indispensable for the initialization of numerical weather prediction (NWP) forecast. To enable the application of observations in NWP models a technical preprocessing is necessary. Within the framework of RC LACE (Regional Cooperation for Limited Area modelling in Central Europe) consortium, a common observation preprocessing system (OPLACE) has been built up to deliver meteorological observations in an appropriate format for data assimilation in the NWP system ALADIN (Air Limiteée Adaptation Dynamique Développment International) The purpose of this paper is to document the OPLACE data sources, preprocessing steps and means to make preprocessed observations available. Furthermore, it describes an exchange of dense national surface synoptic measurements and high-resolution aircraft data in real-time among RC LACE national meteorological services (NMS) of Austria, Croatia, the Czech Republic, Hungary, Romania, Slovakia, and Slovenia.

1 Introduction

Numerical weather prediction (NWP) is an initial-value problem where numerical forecast is obtained by integration of the governing equations of geophysical fluid dynamics from initial values of the meteorological fields (Morel, 1981). Information about the state of the atmosphere is given by meteorological observations, such as surface observations from land and ship stations, upper-air observations from radiosondes and aircraft, and remote sensing observations from satellites and weather radars. Despite of many data sources, the observations are not available in sufficent quantity and with sufficient accuracy. The observations have to be combined with a NWP forecast through the process of data assimilation to provide the initial values for NWP (Kalnay, 2003; Rabier, 2005).

The World Meteorological Organization (WMO) World Weather Watch (WWW) programme ensures the provision and exchange of meteorological observations to support research and operational work in area of weather forecasting. A large variety of observation platforms involves various data formats (WMO, 2015), such as Traditional Alphanumeric Codes (TAC), Binary Universal Form for the Representation of meteorological data (BUFR), and General Representation of fields In Binary (GRIB). In order to facilitate the use of observations in operational NWP, a preprocessing step is needed to treat the measured data. The preprocessing step usually consists of a real time data acquisition and decoding. There might be further steps involved, such as data storage, quality control, and additional format conversions.

NWP is time-critical and computationally demanding. Preprocessing a maximum number of observations is a vital precondition for data assimilation. Therefore, a maximum number of observations should be acquired and transformed to NWP suitable formats, as quickly as possible.

Despite of a long and fruitful collaboration¹ on limited-area NWP within the Regional Cooperation for Limited Area modelling in Central Europe (RC LACE; Wang et al., 2018) the observation preprocessing was originally a responsibility of each National Meteorogical Service (NMS). Consequently data assimilation was implemented rather slowly, e.g. the three-dimensional variational data assimilation (3D-Var; Bölöni, 2006) became operational in Hungary in 2005, surface analysis based on optimal interpolation was implemented in Czech Republic and Hungary in 2006 and 2008, respectively. In order to support operational data assimilation within RC LACE Programme, an internal project was proposed in 2008. The core component was a development of a common system to preprocess meteorological observations for RC LACE (OPLACE). The OPLACE system has been aimed to:

• use limited staff resources efficiently and to reduce computational and network communication costs;

• to provide preprocessed meteorological observations in an appropriate format for data assimilation in ALADIN system (Termonia et al., 2018). Historically the ALADIN system supported a single proprietary ASCII format (OBSOUL). Other formats such as BUFR and GRIB has been implemented progressively;

• to provide a platform for exchange of high-resolution observations, such as dense synoptic networks, and radar data, which are not covered by existing WMO programmes. Such observations will be referred to as the local observations hereafter.

Finally it turned out that OPLACE was a key ingredient for the successful implementation of the data assimilation in local NWP suites of RC LACE countries (Fig. 1) and it has ensured a reliable base for operational NWP purposes.

Figure 1 Milestones of operational data assimilation (DA) implementation in RC LACE countries.

This paper is organized as follows. The OPLACE system, its data sources, processing techniques, and outputs will be described in Sect. 2. Section 2.4 is dedicated to the real time exchange of the local observations. Finally conclusions and a short outlook will be given in Sect. 3.

2 The OPLACE system

The OPLACE system has been hosted by Hungarian Meteorological Service (HMS) since 2009. It was based on already existing infrastructure of HMS, e.g. data acquisition, databases of conventional observations, and satellite data processing. The observations are acquired from different sources and are subject to decoding, format conversions, eventually a simple quality control.

2.1 Data source and inputs

At the time of writing OPLACE provides observations primarily over Europe which has been the area of interest for RC LACE. Conventional observations, e.g., surface synoptic data from land (SYNOP) and ships (SHIP), moored and drifting buoys (BUOY), upper-air sounding (TEMP), wind profiler (E-PROFILE), and aircraft (AMDAR, ACARS) observations, are received via WMO Global Telecommunication System (GTS) in the standard WMO data formats (TAC and BUFR).

Remote sensing² data, such as microwave and infrared observations, atmospheric motion vectors, and scatterometer retrievals, are collected by EUMETSAT’s dissemination mechanism for the near real-time delivery of data (EUMETCast) from geostationary Meteorological Satellite (Meteosat), from polar orbiting Meteorological Operational (MetOp) and National Oceanic and Atmospheric Administration (NOAA) satellites. The data provided by EUMETCast are processed up to different levels before their distribution (WMO, 2019):

• SEVIRI data are retrieved as Level 1.5, i.e., instrument data (digital counts) with geolocation and calibration information;

• AMSU-A/B, MHS, HIRS, ATMS, and IASI data are retrieved as Level 1c, i.e., instrument data with geolocation information and converted to brightness temperature or reflectance factor;

• ASCAT scatterometer retrievals as Level 2, i.e., instrument data converted to geophysical quantity (ocean winds).

Most of satellite data are fetched in BUFR format except for SEVIRI data which are retrieved in High Rate Information Transmission (HRIT) standard used by satellite operators for the dissemination of data to users via direct broadcast.

2.2 Processing technique

A level of processing may differ for a given observation or a group of observations. The central task of the preprocessing is the data conversion to appropriate format for data assimilation in the ALADIN system. Particularities of various observation types are discussed and a short description of general processing tasks is given.

2.2.1 Conventional observations

The conventional observations (except for wind profiler data) are decoded by HMS observation team using locally developed decoders. Selected information is stored in local Network Common Data Form (netCDF) databases. These databases are not used by OPLACE exclusively but serve as inputs for other tools and local applications at HMS, e.g., visualization and verification. Incoming GTS data, in particular SYNOP and TEMP, may be delayed, repeated or corrected. The observations should be processed correspondingly, e.g. the corrections overwrite previous values in the databases.

Despite of a good progress in the WMO migration (WMO, 2002) from the TAC to the Table Driven Code Forms (TDCF) many SYNOP and TEMP data are still received in both TAC and BUFR formats. Both streams are considered in the processing and duplications are eliminated.

The data conversion to appropriate format for data assimilation in the ALADIN system is performed by the OULAN program developed at Météo France. The program has been designed to read observations from a local database and to write the data in the proprietary ASCII format called OBSOUL. Originally all observation types were processed by the OULAN program, but progressively more data formats, e.g., BUFR, netCDF, are supported by the ALADIN system so the OULAN may become obsolete in the future.

The OULAN program has been adapted at HMS to use the local netCDF databases of conventional data. A very simple quality control is performed within the OULAN program. The program rejects observations with undefined identification, date or altitude, when position is missing or out of sphere and when wind components cannot be deduced from reported wind speed and direction (in case of a weak and variable wind). Additionally, the relative and specific humidity are computed from temperature and dew point temperature (if not available in the original report). The OULAN program supports a possibility to employ an additional data rejection based on external quality controls flags, but this has not been used within OPLACE implementation. The processing chain of SYNOP data is illustrated in Fig. 2a.

Figure 2 Illustration of observation preprocessing steps for SYNOP (a) and SEVIRI (b) data.

2.2.2 Remote sensing observations

Historically remote sensing observations were processed by the OULAN as well, and there was an additional preprocessing step of EUMETSAT ATOVS and AVHRR Pre-processing Package (AAPP) involved in format conversions. Currently there is no specific preprocessing of AMSU-A/B, MHS, HIRS, ATMS, and IASI Level 1c data which can be handled by the ALADIN system directly as well as ASCAT Level 2 data.

Atmospheric motion vectors (AMV), derived by tracking the motion of clouds and other atmospheric constituents, can be obtained either as a final product (GEOWIND) from Meteosat Second Generation (MSG) satellites via EUMETCast (EUMETSAT, 2015), or a more high-resolution set of AMVs from MSG (HRWIND) can be obtained by EUMETSAT Satellite Application Facility on support to Nowcasting and Very short range forecasting (NWC SAF; EUMETSAT, 2013). The NWC SAF version 2013 has been used for production of HRWIND observations. Both AMVs products are in BUFR format supported by the ALADIN system, and AMVs do not require any additional format conversion.

The SEVIRI data are processed specifically to allow assimilation of selected channels in clear sky conditions or above very low or mid-level clouds (Montmerle et al., 2007). The cloud-type product developed by CMS (Centre de Météorologie Spatiale, Lannion, France) in the framework of NWC SAF has been used to derive cloud type information (Derrien et al., 2005). The preprocessing comprises the conversion of selected infrared and water vapour channels from SEVIRI instrument to brightness temperatures and the format conversion from HRIT to GRIB format. The SEVIRI data and auxiliary NWP data, e.g. 3 h forecast from European Centre for Medium-Range Weather Forecasts (ECMWF), serve as input to NWC SAF to determine cloud mask and geolocation information. Furthermore, a dedicated program is necessary to generate date and time information from the beginning of the scan. All intermediate information is finally converted into a single GRIB file (Fig. 2b).

2.2.3 General processing

Apart from observation specific procedures and format conversions described above there are general tasks, such as data handling (fetching, partition, uploading) and monitoring. Processed observations are split in hourly time-slots (defined as ±30 min interval of the given hour) and separated by observation type and file format. The separation gives users a flexibility to get data according their needs. Taking into account various inputs and processing techniques it is important to carefully monitor the whole processing chain. Processing steps are traced in log files and missing data or any issue is reported via e-mail.

2.3 Output and data availability technique

A File Transfer Protocol (FTP) server is used to facilitate the transfer of data. Processed observations are organized in a directory structure corresponding to the separation by hourly time-slots, observation type, and file format. The outputs are kept on the server for 6 d to enable a restart of operational applications in case of a short hardware or software malfunction. An archive is not part of the system and it is up to the users to store the data permanently. The OPLACE system runs regularly (as of today every twenty minutes) to update fresh or delayed observations to the FTP server. Table 1 presents an actual overview of available observations and file formats.

Table 1 Overview of observation types, satellite sensors and data formats available in OPLACE.

2.4 Exchange of the local observations

High-resolution observations are important for initialization of NWP at convection-permitting scales but they are not always covered by existing WMO programmes. For example modern air traffic surveillance systems (Mode-S radars) have received substantial attention in recent years due to its capability to provide not only an accurate knowledge of the position of the aircraft, but also meteorological information (de Haan, 2011; Strajnar, 2012). Another substantial number of local observations exists in dense regional synoptic or geodetic networks.

Figure 3 The geographical distribution of (a) surface observations available in the GTS (blue) and of denser national observations (red) and (b) high-resolution aircraft Mode-S EHS data from KNMI (blue) and Mode-S MRAR data from Slovenia (red) and Czech Republic (green).

An extension of OPLACE was proposed to support the exchange of the local observations for operational NWP. Given that acquisition and preprocessing of the local observations can be very complex, see de Haan (2011) for Mode-S data, we have proposed to exchange data that have already been preprocessed as much as possible. The OPLACE system then provides a platform (FTP server) for the exchange, but it can eventually perform format conversions and the splitting into hourly time-slots.

Dense national surface synoptic measurements have been exchanged within RC LACE since 2014, recently the national synoptic data from Poland were added (Fig. 3a). The exchange of high-resolution aircraft Mode-S MRAR (Meteorological Routine Air Report) data from Slovenia became operational in 2015 and has been extended by Mode-S EHS (Enhanced Surveillance) from the Royal Netherlands Meteorological Institute (KNMI) in 2016. Further extension by Mode-S MRAR data from Czech Republic is under preparation (Fig. 3b).

3 Conclusions

The aim of this paper was to describe the OPLACE system which delivers observations in an appropriate format for data assimilation in the NWP system ALADIN. Details of data acquisition and processing techniques were given to demonstrate the complexity of such observation preprocessing. OPLACE has been developed within the RC LACE consortium and is a good example how NMSs can work together in an effective way and save manpower. Furthermore, OPLACE provides a platform for exchange of the local observations not available in the GTS, such as high-resolution surface synoptic and aircraft Mode-S measurements.

In the near future, OPLACE will add observations from new satellite sensors such as CrIS. Other data sources, such as GNSS zenith total delay, radar data, and further Mode-S data are under evaluation and might become available through the OPLACE local observation exchange in the future.

Furthermore, more observations bring increasing data volumes and complexity. The first concept of the OPLACE system from 2008 comprised only a limited number of processing tasks which run sequentially from the KornShell script. Over the years the performance has become an issue, in particular the sequential execution. Therefore a redesign of the system was initiated to allow a parallel execution in a controlled state-of-the-art working environment and to improve the system operations and monitoring.