Due to the rapidly changing climate in the polar regions, the thawing of permafrost soil leads to disturbances like retrogressive thaw slumps (RTS). As these disturbances can further destabilize permafrost and cause damage to existing infrastructure, they require continuous monitoring. They often appear in clusters and are comparably small in size, rarely exceeding 10 ha in size, thus calling for high resolution remote sensing data. Recently launched satellite platforms and publically available datasets (e.g. ArcticDEM) enable this high resolution monitoring from space. However, the vast extent of permafrost areas renders manual analysis infeasible and calls for an automated approach to mapping these disturbances. We present such an automated approach using deep learning.
As deep neural networks generally require large annotated training datasets, we first create a dataset of manually annotated thaw slumps on PlanetScope satellite imagery. This dataset contains imagery from 2018 and 2019 and spans six areas of interest in Canada (Banks Island, Herschel Island, Horton Delta, Tuktoyaktuk Peninsula) and Russia (Kolguev Island, Lena River). The overall spatial extent of the dataset is around 900km² and covers 2172 individual thaw slumps. In extensive pre-processing steps, the PlanetScope optical imagery was enriched by ArcticDEM-derived topography data like relative elevation and slope, as well as tasseled cap time series trends derived from Landsat imagery. Finally, the data is cut into smaller spatial patches to facilitate the training of deep neural networks.
Using this dataset as a basis, we explore deep learning approaches to automated RTS mapping. The task of pixel-wise classification is well researched in computer vision, therefore our approach builds on existing frameworks for semantic segmentation. However, these models were built on assumptions from a different field, with some expecting readily pre-trained backbone networks. Therefore, it is not straight-forward to predict which model will perform best at RTS mapping, calling for a careful evaluation of the models on our dataset. To this end, various configurations of the UNet, UNet++ and DeepLabv3 segmentation models are trained and evaluated. Given the spatial sparsity of the target features, combinations of two training protocols are examined. In sparse training, the model is trained only on patches that contain target features, while in full training, the model is trained on all patches. In our experiments, the best performing combination was a long phase of sparse training followed by a short phase of full training.
In order to evaluate the general performance and spatial transferability of the models, we perform a regional cross-validation on the six study sites, where 5 are used for training and the remaining site is used to evaluate the model performance. The observed model accuracy is promising in some of the validation regions (e.g. Lena, Horton, Kolguev), while others still show room for improvement (Banks Island, Tuktoyaktuk). This can be attributed due to the large variance in geomorphological features between the evaluated regions, where some are easier to generalize to, while others pose a harder challenge for the models.

The automatic detection and tracking of mesoscale ocean eddies, the ‘weather of the ocean’, is a well-known task in oceanography. These eddies have horizontal scales from 10 km up to 100 km and above. They transport water mass, heat, nutrition, and carbon and have been identified as hot spots of biological activity. Monitoring eddies is therefore of interest among others to marine biologists and fishery.

Recent advances in satellite-based observation for oceanography such as sea surface height (SSH) and sea surface temperature (SST) result in a large supply of different data products in which eddies are visible. In radar altimetry observations are acquired with repeat cycles between 10 and 35 days and cross-track spacing of a few 10 km to a few 100 km. Therefore, ocean eddies are clearly visible but typically covered by only one ground track. In addition, due to their motion, eddies are difficult to reconstruct, which makes creating detailed maps of the ocean with a high temporal resolution a challenge. In general, they are considered a perturbation, and their influence on altimetry data is difficult to determine, which is especially limiting for the determination of an accurate time-averaged dynamic topography of the ocean.

Due to their spatio-temporal dynamic behavior the identification and tracking are challenging. There is a number of methods that have been developed to identify and track eddies in gridded maps of sea surface height derived from multi-mission data sets. However, these procedures have shortcomings since the gridding process removes information that is valuable in achieving more accurate results.

Therefore, in the project EDDY carried out at the University of Bonn we intend to use ground track data from satellite altimetry and - as a long-term goal - additional remote sensing data such as SST, optical imagery, as well as statistical information from model outputs. The combination of the data will serve as a basis for a multi-modal deep learning algorithm. In detail, we will utilize transformers, a deep neural network architecture, that originates from the field of Natural Language Processing (NLP) and became popular in recent years in the field of computer vision. This method shows promising results in terms of understanding temporal and spatial information, which is essential in detecting and tracking highly dynamic eddies.

In this presentation, we introduce the deep neural network used in the EDDY project and show the results based on gridded data sets for the Gulf stream area for the period 2017 and first results of single-track eddy identification in the region.

It is clear that extreme tropical cyclone systems are increasing over the warm tropical oceans of the globe, which particularly constitutes a serious concern in the coastal locations with major population density areas. A few analyses have previously been conducted to explore the intensity with other hurricane-related parameters. However, little progress has been made to implement such characteristics in an automated fashion to automatically learn non-linear relationships between physical variables at different time windows. A novel hybrid framework based on supervised and unsupervised machine learning (ML) techniques is presented here to identify and optimally combine a mixed set of early predictors to forecast the evolution of the system, especially those systems attaining the highest categories. The size and brightness temperature of a tropical storm or hurricane are some of the key parameters from Earth Observation data that significantly contribute to better forecasting the intensity of a tropical cyclone system, including tropical storms, category 1-2 (minor hurricanes) and category 3-5 (major hurricanes) hurricane-strength systems. An 80%-20% train-test split was applied for evaluating our ML algorithms over the period from 1995 to 2019 in both the North Atlantic and East Pacific Oceans. Several metrics were also examined to assess robustness of our forecasting models, such as the accuracy rate (AR) or the Cohen’s Kappa value ( 𝜅 ). The hurricane intensity forecast accuracy is tied to the forecast window, being a 16- to 40-hour time window the most accurate one with about 70% AR and 𝜅 value of at least 0.4 (from moderate to perfect agreement). However, our models can also provide reasonable predictions of a hurricane’s intensity up to 56 hours ahead. Overall, a slightly higher intensity forecasting performance was found over the Atlantic Ocean. This promising framework is designed to include additional parameters and classes where relevant for further study.

Solar-Induced chlorophyll Fluorescence (SIF) is a crucial parameter for Earth Observation, as it is strictly related to the vegetation health status. In particular, SIF can be easily monitored through optical remote sensing and offers unique information about vegetation functional state. In fact, SIF emission is one of the three main pathways exploited by vegetation to convert the Absorbed Photosynthetic Active Radiation (APAR) and it is related to pivotal parameters such as the Gross Primary Productivity (GPP), which is a key-role parameter in many environmental applications.
In this contribution, we present a novel approach based on an optimized multi-parameter retrieval algorithm to improve the understanding of the complex relationships between fluorescence and biophysical/biochemical variables. The proposed algorithm is specifically aimed to analyse the reflectance spectra acquired from the vegetation and consistently retrieve the Top Of canopy SIF spectrum, the SIF spectrum corrected for leaf/canopy reabsorption (i.e. at photosystem level), the quantum efficiency (SIFqe) and three canopy-related biophysical parameters (Leaf Area Index - LAI, Chlorophyll content - Cab and APAR) in few milliseconds. This algorithm mainly consists on a novel hybrid Phasor-Machine Learning based approach, which is exploited for the first time for quantitative retrievals in the context of remote sensing studies, and it is the first method capable to retrieve the full fluorescence spectrum at the photosystem level and the fluorescence quantum yield from experimental measurements acquired onsite.
More in detail, our approach exploits reflectance spectra, which are discrete-Fourier transformed on consecutive spectrally resolved complex planes. In each considered complex plane, the spectra which are characterized by the same biophysical and SIF parameters are projected in the same point. It is therefore possible to predict the unknown properties of a single reflectance spectrum by evaluating its projection position in each plane. In order to exploit, for each estimated parameter, the most suitable spectral windows and at the same time, avoiding possible superposition effects in some planes, the algorithm employs a supervised Machine Learning algorithm, trained with the atmosphere-canopy radiative transfer (RT) SCOPE model, which analyses the projection position of the reflectance spectrum in all the considered planes and estimates the investigated variables.
The algorithm has been validated by means of RT simulations, characterizing its retrieving accuracy by varying different parameters (spectral windows width, number of exploited phasor planes, size of the dataset, etc.) and by applying to the analysed spectra an increasing Poissonian noise, described in terms of signal to noise ratio (SNR). Considering the experimental conditions (SNR >= 500), the algorithm is able to independently estimate each biophysical parameter and SIF spectrum with a relative root mean square error (RRMSE) lower than 5%.
In order to investigate the seasonal and daily dynamics of SIF, LAI, Cab, SIFqe and APAR, the method has been also applied to field experimental data collected in the context of the AtmoFLEX and FLEXSense ESA campaigns. Field data were acquired in Grosseto (Italy) from two different crops (forage and Alfa Alfa) by means of the FLOX spectrometer accommodated on ground at top-of-canopy level and on high towers (~100 meters) in a deciduous forest of Downy Oak (France).
The retrieved annual dynamic for SIF spectra has been then compared with the results obtained by state of the art inversion-based methods, showing a good consistency between the two different approaches (RRMSE ~ 10%). Moreover, also the daily dynamics of the investigated variables behave accordingly to the results of the theoretical models.
The retrieval of SIF at the high tower has been investigated excluding the O2 spectral bands affected by the atmospheric reabsorption. The obtained results are promising and it has implications on tower-based measurements, where complex and computationally expensive atmospheric compensation techniques are needed to retrieve fluorescence from oxygen absorptions bands.
In summary, in this study we will show the results of the performance of this new algorithm and its ability in deriving accurate fluorescence spectrum corrected for reabsorption and fluorescence quantum yield from model simulation and real measurements collected over two crops and on a deciduous forest. Overall, the obtained results are of undoubted use and exploitation for the ESA Earth Explorer FLEX Mission, and this study demonstrates a promising potential to exploit ground and tower spectral measurements with advanced processing algorithms, for improving our understanding on the link between canopy structure and physiological functioning of plants and it can be straightforwardly employed to process reflectance spectra to open new perspectives in fluorescence retrieval at different scales.

Complex numerical weather prediction (NWP) models are deployed operationally to predict the future state of the atmosphere. While these models solve numerically a system of partial differential equations based on physical laws, they are computationally very expensive. Recently, the potential of deep neural networks has been explored in a couple of scientific studies to generate bespoken weather forecasts for short time scales, inspired by the success of video frame prediction models in computer vision. In this presentation, we explore the application of different deep learning networks from the field of video prediction for weather forecasts up to 12 hours and discuss the potential and limitations of these approaches.

In the first study, we focus on the predictability of the diurnal cycle of near-surface temperatures. A ConvLSTM, and an advanced generative network, the Stochastic Adversarial Video Prediction (SAVP), are applied to forecast the 2 m temperature for the next 12 hours over Europe. Results show that SAVP is significantly superior to the ConvLSTM model in terms of several evaluation metrics. Our study also investigates the sensitivity to the input data in terms of selected predictors, domain size and number of training samples. The results demonstrate that additional predictors, i.e. in our case the total cloud cover and the 850 hPa temperature, enhance the forecast quality. The model can also benefit from a larger spatial domain. By contrast, the effect of reducing the training dataset length from 11 to 8 years is rather small. Furthermore, we reveal a small trade-off between the MSE and the spatial variability of the forecasts when tuning the weight of the L1-loss component in the SAVP model.

In the second study, we explore a custom-tailored GAN-based architecture for precipitation nowcasting. We developed a novel method named Convolutional Long-short term memory Generative Adversarial Network (CLGAN), to improve the nowcasting skills of heavy rain events with deep neural networks. The model constitutes a GAN architecture whose generator is built upon an u-shaped encoder-decoder network (U-Net) equipped with recurrent LSTM cells to capture spatio-temporal features. A comprehensive comparison between CLGAN, the advanced video prediction model PredRNN-v2 and the optical flow model DenseRotation is performed. We show that CLGAN outperforms in terms of point-by-point metrics as well as scores for dichotomous events and object-based diagnostics. The results encourage future work based on the proposed CLGAN architecture to further improve the accuracy of precipitation nowcasting systems.

In atmospheric remote sensing, the quantities of interest (e.g. the composition of the atmosphere) are usually not directly observable but can only be inferred indirectly via the measured spectra. To solve these inverse problems, retrieval algorithms are applied that usually depend on complex physical models, so-called radiative transfer models (RTMs). These are very accurate, however also computationally very expensive and therefore often not feasible in combination with the strict time requirements of operational processing. With the recent advances in machine learning, the methods of this field, in particular deep neural networks (DNNs), have become very interesting in order to accelerate and improve the classical remote sensing retrieval algorithms. However, their application is not straightforward as they can be used in different ways and there are many aspects to consider as well as parameters to be optimized in order to achieve satisfying results.

For the inverse problem in atmospheric remote sensing, there are two main approaches to apply neural networks:

1. Neural networks for solving the direct problem, where a neural network approximates the radiative transfer model and can thus replace it as a forward model for the inversion algorithm
2. Neural networks for solving the inverse problem, where a neural network is trained to infer the atmospheric parameters from the measurement (and some additional information, e.g. surface properties, viewing geometry) directly

For the first case, we present a general framework for replacing the RTM with a DNN that offers sufficient accuracy while at the same time increases the processing performance by several orders of magnitude. It's application is shown at the example of the ROCINN algorithm which is used for the operational cloud product of the copernicus satellites Sentinel-5 Precursor (S5P) and Sentinel-4 (S4).

The second case is more challenging: In contrast to approximating the radiative transfer model which provides a well defined function from the parameters to the measurements, the inverse problem is ill-posed. Due to this, small differences in the measurements can lead to large differences in the retrieved quantities. It is therefore desirable, to characterize the retrieved values by an estimate of uncertainty describing a range of values that are probable to produce the observed measurement. This can be achieved by using a bayesian framework, like it is done in bayesian neural networks or more novel methods like invertible neural networks. Based on these techniques, we show first results of the retrieval using neural networks for solving the inverse problem.

Finally, we will compare the two different approaches of using DNNs for the retrieval of cloud properties for S5P / S4 and discuss their applicability for the future.