Earth observation satellites are used for many tasks, among them the monitoring of agricultural areas. For example subsidence farming can be monitored to predict food shortages, which in turn helps organizing humanitarian aid faster when there is a shortage. Often, optical data is used because of the easy interpretability and especially the normalized difference vegetation index (NDVI) is frequently used for vegetation monitoring. However, in tropical areas with frequent cloud coverage or subtropical areas where the main growing season is during the rainy season clouds hinder the acquisition of optical images. To avoid this, active cloud-penetrating sensors like synthetic aperture radar (SAR) can be used. However, the greatly different characteristics of SAR images make interpretation more difficult and usable intelligence is harder to obtain.
There is great demand to mitigate this problem by converting SAR backscatter values to artificial NDVI values, which then in turn can be used for downstream tasks. This idea was already demonstrated in two studies [1, 2]. However, these studies are limited to small areas and present conversion models that are not globally applicable. Additionally, they suffer from a low performance when only relying on backscatter values and not using additionally data sources like the last cloud free NDVI value.
As solution we present a globally applicable model for the conversion of SAR backscatter values to NDVI values using a deep neural network. The used model does not rely on optical data at application time and is therefore unaffected by cloud cover.

To train the model, a dataset consisting of Sentinel-1 SAR data and Sentinel-2 optical data is created. To have a direct relation between backscatter and NDVI values the temporal distance is at most 12 hours between images of the same area. This avoids other influences like seasonal changes or vegetation growth. Images were sampled globally with an equal distribution for climate zones and land covers to capture the full spectrum of earth surfaces and vegetation. As auxiliary data, the 10m resolution ESA WorldCover product [3] and the 30m resolution ALOS JAXA DEM [4] were retrieved. Google Earth Engine was used to download the data.
The used model is a slightly adapted UNet. It does a pixel-wise regression of the NDVI using the VV and VH polarizations of the Sentinel-1 data, the ESA WorldCover and the ALOS DEM.

Using this approach, a globally applicable model is created to predict the NDVI from cloud-penetrating SAR images. This removes the need to train models for specific regions and vegetation. One disadvantage of this approach is the lower resolution of Sentinel-1 images (with a pixel size of 20x22m [5]) compared to the 10x10m resolution of Sentinel-2 images. This prevents the correct prediction of some fine spatial details. Further research is needed to increase the resolution regarding those details, either by using time series as input instead of images of a single date or by using other data sources to include more structural details.



ACKNOWLEDGMENT

This work was supported by the German Federal Ministry for Economic Affairs and Energy in the project “DESTSAM - Dense Satellite Time Series for Agricultural Monitoring” (FKZ 50EE2018A).


REFERENCES

[1] G. Scarpa, M. Gargiulo, A. Mazza, and R. Gaetano, “A CNN-based fusion method for feature extractionfrom sentinel data,” Remote Sensing, vol. 10, no. 2, Art no. 236, 2018.
[2] R. Filgueiras, E. C. Mantovani, D. Althoff, E. I. F. Filho, and F. F. da Cunha, “Crop NDVI monitoring based on Sentinel 1,” Remote Sensing, vol. 11, no. 12, Art no. 1441, 2019.
[3] D. Zanaga, R. Van De Kerchove, W. De Keersmaecker, N. Souverijns, C. Brockmann, R. Quast, J. Wevers, A. Grosu, A. Paccini, S. Vergnaud, O. Cartus, M. Santoro, S. Fritz, I. Georgieva, M. Lesiv, S. Carter, M. Herold, Linlin Li, N. E. Tsendbazar, F. Ramoino, O. Arino, ”ESA WorldCover 10 m 2020 v100,” 2021.
[4] J. Takaku, T. Tadono, M. Doutsu, F. Ohgushi, and H. Kai, “Updates of ‘AW3D30’ ALOS Global Digital Surface Model with Other Open Access Datasets”, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS, vol. XLIII-B4-2020, pp. 183–189, 2020.
[5] Collecte Localisation Satellites, ”Sentinel-1 product definition,” Online, 2015.

The monitoring of the ice cover is important primarily for navigation, but also to study the water cycle and surface energy flux ; snow and ice cover on lakes is one of the 50 ECVs (Essential Climate Variables). The European Environment Agency (EEA) recently released near real time (NRT) snow and ice classification products over the EEA38 + UK European zone, using Sentinel-2 imagery as well as Sentinel-1 SAR data, with a decametric resolution : https://land.copernicus.eu/pan-european/biophysical-parameters/high-resolution-snow-and-ice-monitoring
The two main NRT products, derived from Sentinel-2 L2A MAJA images (https://logiciels.cnes.fr/fr/content/maja) are the Fractional Snow Cover (FSC) indicating the fraction of snow on 20m pixels, and the River and Lake Ice Extent (RLIE), also at 20m resolution, indicating the presence of ice / water / other on the river and lake Eu-Hydro mask. Barrou Dumont et al. (2021) show a very good match between in-situ data and the FSC products, and Kubicki et al. (2020) also show a good performance for RLIE products through comparisons with in-situ data as well as very high resolution SPOT 6/7 and Pléiades 1A/1B images. However, the RLIE validation results were obtained mainly on northern regions during winter, and were later shown to present a large number of ice classification false positives on turbid waters and salt lakes. This presents a problem both for climate analysis because of the large number of ice false positives during summer on turbid waters, and for the generalization of the algorithm to the full globe. Thin ice or black (transparent crystalline) ice also often goes undetected by the RLIE algorithm. Furthermore, the RLIE product requires the EU-Hydro water mask which is static and therefore limits the ability to track variability in river beds or lake surfaces, and misses a large number of small rivers and lakes not represented in the EU-Hydro water mask.
The RLIE algorithm relies on a minimum distance classifier as well as specific thresholds on certain bands to reduce false positives on turbid waters. We show that using more capable machine learning and deep learning methods, we are able to almost completely remove ice false positives on salt lakes and turbid waters, considerably improve thin ice and black ice detection, as well as generate products on the full images without the need for an a priori water mask.
Machine learning and deep learning results were obtained on the basis of 32 fully labelled Sentinel-2 images covering different regions on the globe, representing various cases including ice melt, ice formation over river and lakes, salt lakes, urban areas, fields, forests, mountain regions, as well as various turbid water cases.
For the machine learning approach, pixels are classified using solely the spectral information on the different Sentinel-2 L2A bands as well as various normalized difference indexes (NDSI, NDVI, NDWI, ...). Both the linear SVM and the RandomForest methods were evaluated : SVM yields better results as RandomForest has a tendency to overfit the input data and produce noisier results. The classification using those methods is considerably improved when compared to the RLIE product. Ice false positives on salt lakes and turbid waters are almost completely removed, but there are still some ice false positives, mostly isolated pixels, which yields somewhat noisy results in some cases. Thin ice / black ice classification is also much improved when compared to RLIE but in some cases the ice still goes undetected although a visual inspection of the image by a human operator clearly recognises the ice. This is due to our ability to use spatial pattern recognition and exploit water / ice borders as well as cracks in the ice.
To improve on the machine learning classification, we used deep learning to exploit spatial information. Two neural networks were evaluated: EfficientNet + DeepLabV3+ as well as EfficientNet + RefineNet. Preliminary results at the date of this abstract indicate that the deep learning approach effectively reduces noise in the classification of products, completely removes ice false positives over turbid waters and adequately classifies ice that was missed in the machine learning approach despite having very visible cracks and water/ice borders. However, in many cases, the borders between categories are less accurate than using the machine learning approach, and manual setting of weights in the cost function is necessary to fine tune the algorithm and avoid over or under classification of the less represented categories in the database (salt lakes are the less represented). Those results are however preliminary and will be consolidated in the coming months.

Barrou Dumont, Z., Gascoin, S., Hagolle, O., Ablain, M., Jugier, R., Salgues, G., Marti, F., Dupuis, A., Dumont, M., and Morin, S.: Brief communication: Evaluation of the snow cover detection in the Copernicus High Resolution Snow & Ice Monitoring Service, The Cryosphere, 15, 4975–4980, https://doi.org/10.5194/tc-15-4975-2021, 2021.

Kubicki, M., Bijak, W., Banaszek, M., Jasiak, P., High-Resolution Snow & Ice Monitoring of the Copernicus Land Monitoring Service Quality Assessment Report for Sentinel-2 ice products, 2020 : https://land.copernicus.eu/user-corner/technical-library/hrsi-ice-qar

There is no doubt that the devastating socio-economic impacts of floods have increased during the last decades. According to the International Disaster Database (EM-DAT), floods represent the most frequent and most impacting event among the weather-related disasters regarding the number of people affected. Nearly 1 billion people were affected by inundations in the decade 2006–2015, while the overall economic damage is estimated to be more than $300 billion, with individual extreme events, like Superstorm Sandy, costing several tens of billions in damage (an estimated $65 billion in damages in the US). Despite this evidence and the awareness of the environmental role of rivers and their inundation, our capability to respond to and forecast floods remains very poor, mainly due to the lack of measurements and ancillary data at large scales.
In this context, satellite sensors represent a precious source of observation data that could fill many of the gaps at the global level, especially in remote areas and developing countries. With the proliferation of more satellite data and the advent of ESA's operational Sentinel missions under the European Commission's Copernicus Programme, satellite images, particularly SAR, have been assisting flood disaster mitigation, response, and recovery operations globally. In addition, the proliferation of open satellite data has advanced the integration of remotely sensed variables with flood modeling, which promises to improve our process understanding and forecasting considerably.
In recent years, the scientific community has shown how earth observation can play a crucial role in calibrating and validating hydraulic models and providing flood mapping and monitoring applications to assist humanitarian disaster response.
Although the number of state-of-the-art and innovative research studies in those areas is increasing, the full potential of remotely sensed data to enhance flood mapping has yet to be unlocked, especially since the latency issue is not being sufficiently well addressed. Indeed, the time between image acquisition to the flood map delivery to the person who needs it is not in line with disaster response requirements. For instance, at the moment, almost all flood maps reach the field teams after two days of image acquisition, which renders the map pretty much useless for instant rescue operations. A delay of 3 days renders the map unusable for almost all operation stages. While until recently delays appeared unavoidable due to the mapping process not being highly automated by transferrable AI, the lion's share of time loss has now shifted to the communication of requests and data. Indeed, it is now responsible for the slow uptake of EO-based products, such as flood maps, into an operational timeline or disaster response protocol of various potential user organizations, such as the UN World Food Programme.
In this work, we present the conception of a Digital Twin Experiment (DTX) to generate a prototype AI-based solution that could be deployed onboard SAR satellites to produce flood maps in near-real-time.
With our contribution, we aim to conceptualize the processing needed onboard the satellite: from the SAR image formation to the Machine Learning-based flood detector and classifier. On the other hand, we provide a strategy for the fast delivery of the onboard inference to achieve short latency. The mapping results consist of column/row raster vectors indicating the flooded area. Therefore, after the creation of a flood map on the "orbital edge", it will be sent in the form of a short "message" and delivered to the field response teams via satellite communication technology for use within minutes, rather than many hours to days as it is currently the case.
We aim to simulate various scenarios by varying the quality of the input SAR data and experimenting with Machine Learning approaches for image segmentation. We ground our experiments on three different viewpoints. First, we aim to assess the impact of SAR image quality on the final result. Therefore, we simulate with DLR's End-2-End SAR simulator different scenes at different resolution and noise levels. Pre- and post-flood scenarios will be considered. Secondly, we also consider possible approximations in the SAR focusing.
On the one hand, we could suppose that it will be possible to obtain a perfectly focused SAR image onboard and with fast computational time. On the other hand, we could already assess the performance of a non-perfectly focused SAR image on the final classification result. For this purpose, we experiment by approximating the standard focusing kernel of a strip-map acquisition mode. We provide, therefore, a dataset of SAR images with different deformations due to the non-perfect SAR focusing. An example of focusing kernel approximation might be to perform unfocused azimuth processing. As a final aspect, we will consider different deep learning models for semantic segmentation, trained to perform at their best in the different proposed scenarios. Eventually, the performance comparison will provide the best trade-off between achieved classification performance and computational effort of the entire processing chain, including SAR focusing. We eventually propose a holistic evaluation strategy for the proposed end-to-end framework, which will provide highly representative and practically oriented metrics for mapping accuracy and processing efficiency.

Identifying regions that have high likelihood for wildfires is a key component of land and forestry management and disaster preparedness. In recent years there has been considerable interest on wildfire modeling and prediction in the domain of machine/deep learning. Focusing on next day fire risk prediction, we have already conducted a thorough research on this specific problem [1, 2] achieving very promising results and gaining valuable insights into the complexities and specificities of task. In our previous work, we formalized the problem as a binary classification task between the fire/no-fire classes, considering as instances the daily snapshots of a grid with 500m-wide cells. Each instance is represented by earth observational, meteorological and topographical data features derived up until the previous day of the prediction. To this end, we utilize a massive dataset, labeled with fire/no-fire information for each grid cell and in daily granularity, covering the whole Greek territory for the years 2010-2020 (focusing mainly on months April to October, which correspond to the main fire season). In this paper, we discuss the major specificities of the task that we have identified by our work so far and propose a concrete Deep Learning (DL) framework that has the potential to jointly handle them.
0.1 Next day fire prediction specificities
Highly imbalanced dataset. The amount of instances that represent the classes of fire and no-fire follow a different distribution with the respective ratio being of the order ∼1 : 100,000. As a conse- quence, vanilla Machine Learning (ML) algorithms are expected to learn biased classification models, due to their inherent design to optimize based on accuracy-like measures, which are improper for imbalanced datasets [3]. As a consequence, the most important class (representing risk of fire occurrence in our case) is poorly modeled and predicted. Various techniques can be employed to mitigate this issue, including class over/under-sampling and cost-sensitive training (i.e. assigning different
weights to the cost of instances of different classes, during the algorithms training/optimization process) [3]. Although these techniques have been extensively studied in the past [4], applying them as is, without adaptation to the task’s particularities, can become problematic. For example, random under-sampling could exclude valuable information from the dataset, whereas oversampling may introduce undesirable noise to the data by interpolating new points between marginal outliers and inliers (data observations that lie in the interior of a data set class in error). Moreover, these approaches do not eventually model the true data distribution of the problem, thus present limited capability to solve real world problems. Finally, such techniques are methodologically poorly applied, e.g. by also performing over-/under-sampling also on the test setting, resulting to poorly, even erroneously, assessed techniques. Pseudo-negative class (alt. absence of fire). Adding to the above problem of fire instances sparseness, there exists a considerable amount of no-fire instances that are not really negative examples, but in fact they denote the absence of fire. This can be misleading since it does not necessarily mean low fire risk, but lack of fire precursor for a wildfire to start. Hence, these particular instances lie very close to fire instances in the feature space, which makes it harder for the models to be properly trained, since the decision boundaries between these cases cannot be clearly established. Thus, traditional ML models demonstrate poor class separability, mainly in the samples lying close to the decision boundaries. We performed an indicative similarity analysis between the fire/no-fire instances and between the fire instances in subsets of our dataset comprising separate months. The results for
August 2010 are reported in the Tables 1 and 2, where similarity class 1 denotes dissimilar instances and similarity class 10 means very similar instances. The following findings are pointed out:
•More than 70% of the fire instances are very similar (similarity class>= 6) to no-fire instances, while
•The similarity between the fire instances is split among the classes 5 to 9 with class 9 having the highest percentage as expected. It also observed that some fire samples are perfectly aligned (class 10), which is explained by the high spatiotemporal correlations inherit in the data [2].

The similarity measure is based on the calculation of the Euclidean distance between the normalized feature vectors of the instances, and was performed for the 1K most similar samples on a monthly basis. The demonstrated results are representative and referred to August of 2010, as similar patterns were found to all the analyzed months and years.

Concept Drift. Another particularity comprises concept drifts (alt. data shifts) that were detected in some feature cases, between the different months and years. For example, the meteorological parameters fluctuate periodically during the year, but also get altered through the years due to the climate change. A typical drift example of data is presented in Figure 1, where a significant drift of the mean temperature, for both fire and no-fire classes, is observed in the year 2012. Further, this feature has strong fluctuations mainly in the fire class through the years. It also worth noticing that in 2018, the mean temperature of the no-fire cells exceeds the mean temperature of the fire cells. Additionally, although at most years fire and no-fire boxplots present significant range differences (i.e. 2010 and 2013), in 2016 the variation of fire samples is a subset of the no-fires’ dispersion, which makes class separability impossible. These underlying changes in the statistical properties of the
data, could degrade the predictive performance of the ML models.
0.2 Handling next day fire prediction via Siamese Networks
We consider the deployment and adaptation of supervised deep metric learning architectures, like Siamese Neural Networks (SNN) [5], as a promising framework for handling the aforementioned specificities of the next day fire prediction task. A SNN consists of two identical NN architectures which are trained in parallel. One sample is passed to the first network, an other to the second and the system is trained on a distance function between the vector representations produced by the identical networks. The aim is to generate good representations of the data, in order to bring similar samples closer and dissimilar samples further away in the distributed representation space. Various loss functions can be selected for optimization, including triplet loss functions [6], which we believe that have great potential for the specific problem examined in this paper. At each iteration of the training process a baseline (anchor) instance is compared to a positive (same class) and a negative (opposite class) input, and the system tries to minimize the distance between the anchor-positive samples and maximize the distance between anchor-negative samples. This training schema is adjustable to our data and could undertake several modifications, extensions and customizations in order to deal with several of the specificities of the problem. Firstly, in order to deal with the imbalanced dataset and the absence of fire concept, the loss function could be customized to generate the triplets by filtering difficult examples from the majority class instead of randomly select the positive and negative samples. Another approach would consist in the relabeling of the noisy samples of the no-fire class to fire class ones. These actions would not only redefine the class boundaries during training, in order to facilitate the identification of more meaningful class boundaries, but also ameliorate to some extent the extreme class imbalance. Considering concept drift, we have the intuitions that distinct cases of the fire class (rare cases [3]) could be identified and split into distinct classes, corresponding to clusters of fires with different characteristics. Properly adjusting the triplet generation process to create triplets dedicated for each of the different fire classes, could be an additional tool towards learning better (and more specific) representations for the fire instances and, eventually, more accurately handling the problem.
References
[1] Alexis Apostolakis, Stella Girtsou, Charalampos Kontoes, Ioannis Papoutsis, and Michalis Tsoutsos. Implementation of a random forest classifier to examine wildfire predictive modelling in greece using diachronically collected fire occurrence and fire mapping data. In Jakub Lokoc, Tom ́as Skopal, Klaus Schoeffmann, Vasileios Mezaris, Xirong Li, Stefanos Vrochidis, and Ioannis Patras, editors, MultiMedia Modeling - 27th International Conference, MMM 2021, Prague, Czech Republic, June 22-24, 2021, Proceedings, Part II, volume 12573 of Lecture Notes in Computer Science, pages 318–329. Springer, 2021.
[2] Stella Girtsou, Alexis Apostolakis, Giorgos Giannopoulos, and Charalampos Kontoes. A machine learning methodology for next day wildfire prediction. In IGARSS, 2021.
[3] Haibo He and Yunqian Ma. Imbalanced Learning: Foundations, Algorithms, and Applications. Wiley-IEEE Press, 1st edition, 2013.
[4] Qiang Yang and Xindong Wu. 10 challenging problems in data mining research. International Journal of Information Technology Decision Making, 05(04):597–604, Dec 2006.
[5] Vijay Kumar B. G, Gustavo Carneiro, and Ian Reid. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. arXiv:1512.09272 [cs], Aug 2016. arXiv:1512.09272.
[6] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large Scale Online Learning of Image Similarity through Ranking, volume 5524 of Lecture Notes in Computer Science, page 11–14. Springer Berlin Heidelberg, 2009.

In recent years we witnessed an increasing number of remote sensing data being available through public satellite data sources (particularly Copernicus Sentinel and Landsat) and distributed in public cloud infrastructures like Amazon Web Services, Google Cloud and the various DIAS and national mirroring infrastructures throughout Europe. Also this evolution overlaps with the recent technological developments as is the introduction of Cloud Enabled GeoTiff, facilitating easy "data cube" like access to exposed data and also the availability of tools like Stackstac enabling friendly access from common data processing tools like Dask.

Both the available data and tools represent a great opportunity for employing Machine Learning methods on a wider scale, integrating various types of satellite and in-situ observations and opening the door towards new techniques and thematic fields.

In this context, we present an updated version of the Hugin EO Machine Learning tool providing support for STAC (Spatio Temporal Asset Catalog) based data cubes and for emerging technologies like Zarr, XArray and stackstac, providing Cloud Native data access.

One of the advantages of HuginEO is it's interoperability with Jupyter based notebooks, enabling users to easily experiment with new models, visualize predictions and analyze various metrics.

Also, we introduce support for additional backend Machine Learning technologies and model types (eg. self-supervised models), and extended support for hyper-parameter model optimization. All models are tested and evaluated against a number of use cases referring to the exploitation of spatio-temporal and spectral features of satellite data, considering their expected use in agriculture and forestry for monitoring purposes and possible integrations in national, thematic, user-oriented Earth Observation data platforms. The current evaluation of the model performance, considering the state of the art metrics, show promising results.

HuginEO is accompanied with a suite of pretrained models for crop classification, building detection in VHR data, super-resolution and models trained using self-supervised techniques (usable for transfer learning in various other applications).