National Ice Centers around the world use satellite data to produce sea ice information for maritime safety in the Arctic, and due to global warming there is an increased focus on monitoring the sea-ice pack throughout the Polar Regions. In the Ice Centers sea ice charting is based on visual interpretation of satellite imagery, and is carried out by trained ice analysts. In this process the ice analyst manually digitizes sea ice-polygons “on top” of the satellite images. This is a both labor intensive and time consuming process.

For high resolution sea ice charting – applicable for navigation at sea – Synthetic Aperture Radar (SAR) satellite images are the main source of information. This is due to the high spatial resolution and the capability of acquiring images independently of clouds and sun illumination. However, different ice types and concentrations as well as different wind conditions over the ocean have the same SAR backscatter signature, which can lead to ambiguities in the relationship between SAR backscatter and ice conditions. Thus, ice analysts may also include observations from other space borne sensors such as Passive Microwave Radiometer where sea ice and water usually are more easily distinguished. However, PMR is much coarser in spatial resolution than SAR and can therefore not be used “stand alone” for derivation of ice charts that are detailed enough to be used for navigation at sea.

The now former AI4Arctic project did apply deep learning, in particular deep convolutional neural networks, for Earth observation applications within the cryosphere, focusing on sea ice and snow (see also https://eo4society.esa.int/projects/ai4arctic/). A major outcome of AI4Arctic is an extensive training dataset known as the ASID-v2 dataset. The ASID-v2 dataset covers all the waters around Greenland over a period of 14 moths (March 2018 to May 2019). The dataset is designed so that it can be used for training of Convolutional Neural Networks (CNNs) in identifying and classifying sea ice in satellite imagery automatically. The dataset consists of three types of data: 1. 461 calibrated and noise corrected dual-pol (HH, HV) Sentinel-1 SAR EW images, 2. corresponding AMSR2 PMR data (seven frequency channels ranging from 6.9 to 89.0 GHz in both H and V polarization), and 3. the corresponding ice charts that were manually produced by the ice analysts in the Ice Service at the Danish Meteorological Institute (DMI). All 461 SAR scenes are co-located in space and time with PMR imagery and ice charts: I.e.: For each SAR scene the best possible match in space and time between PMR observations and ice charts are found; then the corresponding PMR imagery and ice charts have been rectified to match the spatial geometry of the SAR scenes.

The new AI4Arctic_CCN training dataset, we present here is an extension of the ASID-v2 dataset and will thus be named ASID-v3. The extension will be in both space and time, and with additional parameters: I.e. the dataset will be expanded geographically by including ice charts from the Canadian and Norwegian waters, and we will try to include maps from Russian National Ice Centers. Additionally, the dataset will be extended for Greenland waters by including ice charts that have been produced since the generation of the ASID-v2 dataset in the years 2019, 2020, and 2021. Furthermore, the dataset will be expanded by including wind fields as well as surface and 2-meter air temperatures from ECMWF Reanalysis v5 (ERA5). The ASID-v3 will also include Sentinel-1 IW images in addition to Sentinel-1 EW images, all preprocessed with the newest available calibration- and noise correction schemes.

In summary: The AI4Arctic_CCN/ASID-v3 training dataset will be substantially larger than ASID-v2 and will include a larger variety of data – both in terms geographical coverage of the satellite imagery and met-ocean parameters. This will enhance our ability to train CNNs to recognize and classify sea-ice with better accuracy at both regional and pan Arctic scales.

Dams are common hydraulic structures built across rivers or streams to store water for irrigation, water supply, hydropower production, and flood protection, among others. Several databases exist, including mainly large dams. These databases have been built over the systematic effort of professional dams’ associations or through volunteers’ endeavors. A complete and accurate global database of dams will be essential for future monitoring and safety of such infrastructures. Despite multiple efforts to map dams, existing inventories are still incomplete, especially when considering the inclusion of smaller dams. Maintaining these databases is not automated and involves manual labor.
The abundance of freely available satellite imagery in the last decade, as well as the development of Cloud infrastructures such as Google Earth Engine and Tensor Processing Units (TPUs), have created an opportunity to automate the process of mapping and monitoring of dams.
The goal of this research is to develop an automated tool to detect and segment dams and to apply it globally, thus identifying the missing dams in the existing databases. Despite the limited (10-15 m) spatial resolution of freely available satellite datasets (Landsat and Sentinel), we will demonstrate that dams of different sizes can be identified and segmented with the help of a U-Net style convolutional neural network architecture.
We will present the results of training a model with different types of inputs exported for 1500 dams, where we could identify the dam crest as a polygon. The model was trained with multiple input features, such as statistics of multi-temporal optical and radar satellite imagery, elevation models, and water occurrence. For the training we used TPU, allowing much shorter training times when compared to GPU. For the final model, we will elaborate on the ability of the multi-channel U-Net network (Xception-style) to see different dam features by analyzing activations of the model on different images. The final goal of this research is to apply the trained network globally to complete existing dam datasets.

This research is funded by Google.org Impact Challenge on Climate 2020.

The majority of our planet’s land surface is covered by haze or clouds. Such atmospheric distortions impede the capability of spaceborne optical satellites to reliably and seamlessly record noise-free data of the Earth’s surface. The need for cloud-free earth observation hence gave rise to a rapidly growing number of haze and cloud removal methods. Most previous methods focus on a multi-modal approach. These techniques reconstruct cloud-covered pixels via information translated from synthetic aperture radar (SAR) or other sensors more robust to atmospheric disturbances, yet focus on only a single time-point of observations. In comparison, recent models attempt a temporal reconstruction of cloudy observations by means of inference across time-series, utilizing the circumstance that the extent of cloud coverage over a particular region is variable over time and seasons.


Our submitted work combines both preceding approaches in a unified framework and thus considers the challenge of cloud removal in optical satellite imagery by integrating information across time and within different modalities. For this purpose, we curate a new data set called SEN12MS-CR-TS, which contains multi-temporal and multi-modal satellite observations. Specifically, SEN12MS-CR-TS consists of one year long time-series of co-registered radar Sentinel-1 (S1) as well as multi-spectral Sentinel-2 observations (S2) acquired in a paired manner. The paired observations feature over 50 regions of interest (ROI) from all over the world and cover over 80,000km² of land surface. The ROI are selected such that our data set is backward-compatible with previous data sets containing annotations that can be used for land cover segmentation or land type classification. For every ROI, 30 time intervals are evenly spaced throughout the year of 2018 to obtain a time series of observations. That is, we collect a paired data set of S1 and S2 time series over all contintents of our planet in order to allow for whole-planet and generalizable cloud removal by means of multi-modal a nd multi-temporal methodology. Importantly, our data set contains the full range of cloud coverage: from clear view observations, over filmy or semi-transparent atmospheric disturbances to densely covered clouds. That is, SEN12MS-CR-TS reflects the broad spectrum of cloud coverage that a remote sensing practitioner may encounter in real use-case scenarios.


We highlight the benefits of the proposed data set by training and testing two different models on our data set: First, a multi-modal multi-temporal 3D-Convolution Neural Network that predicts a cloud-free image from a sequence of cloudy optical and radar images. Second, a sequence to sequence translation model that predicts a cloud-free time series from a cloud-covered time series. Both approaches are evaluated experimentally, with their respective models trained and tested on SEN12MS-CR-TS. The considered methods and their performances are systematically analyzed in terms of pixel-wise and image-wise metrics. The conducted experiments highlight the contribution of our curated data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy information.


The main contribution of this work is in curating and providing SEN12MS-CR-TS; a multi-modal multi-temporal dataset for cloud removal in optical satellite imagery. Our large-scale data set covers a heterogeneous set of ROI sampled from all over Earth, acquired in different seasons throughout the year. Given that the contained observations cover clear-view, filmy, as well as non-transparent dense clouds, the objective of reconstructing cloud-covered information poses a challenging task for the considered methods and future approaches. For the sake of demonstrating the usefulness of the presented data set, we propose a sequence to point as well as a sequence to sequence cloud removal network. We provide evidence that taking time series information into account is facilitating the reconstruction of cloudy pixels and that including multi-sensor measurements does further improve the goodness of the cloud-removed predictions, justifying the design of SEN12MS-CR-TS to include multi-temporal and multi-modal data. The conducted experiments highlight the contribution of our curated data set to the remote sensing community as well as the benefits of multi-modal and multi-temporal information to reconstruct noisy satellite information. Given the omnipresence of haze and clouds in spaceborne optical satellite imagery as well as the hindrance it poses for a continuous monitoring of our planet, we hope that our contribution facilitates further directions of the research community to an obstacle-free application of remote sensing.

Machine learning (ML) systems combined with satellite Earth observation (EO) data can be used to develop critical datasets and insights that provide targeted, geography-specific information including early warning of droughts, crop failure, and pests; impact assessment of climate-related disasters such as flooding and landslides; and tracking and responding to population displacement or health emergencies. This information can lead to life-saving decisions, policies, and emergency operations. However, there are currently many challenges for developing ML systems that use EO data, namely, limited public labeled datasets, a lack of harmonization across labels and source data, and substantial effort required to make EO data “ML-ready”. Further, there are many barriers that limit the accessibility of satellite data to ML practitioners, including a lack of large labeled datasets as well as an understanding of the range of satellite products available, how these products should be processed, and how to manage multi-dimensional geospatial data. To lower these barriers and facilitate the use of satellite datasets by the ML community, we present CropHarvest—a satellite dataset of more than 90,000 geographically diverse samples with agricultural class labels.

CropHarvest is a spatially and semantically comprehensive dataset of agricultural class labels. This dataset harmonizes 20 datasets with agricultural class labels, including existing public datasets and new datasets released with this paper. We provide the label geometries as well as the corresponding satellite data inputs from four satellite datasets (Sentinel-2, Sentinel-1, topography, and climate). We additionally provide three benchmark tasks for evaluating the performance of models in a range of agroecologies and dataset-size regimes. We aim to contribute to the Earth Observation Digital Transformation by releasing the CropHarvest data with a python package which mimics the torchvision API. This will enable ML practitioners to easily begin interacting and developing models with the dataset. By including datasets and tasks for data-sparse regions, we hope to enable the development of crop type classification models and other ML systems with improved performance in developing regions that are typically under-represented in ML datasets. The data and accompanying python package are available at https://github.com/nasaharvest/cropharvest.

As known, many EO applications with machine learning techniques, especially deep learning tasks, have raised the importance of datasets. However, data collection may be quite expensive, as it usually costs lots of time and human labor.
Our motivation is to prepare AI-ready Earth observation training datasets at minimal cost.
We try to tackle these questions:
How do we foster the generation of EO image datasets?
How do we guarantee the quality of the generation of EO image datasets?
The answer to the first question is to use active learning techniques, which adds human expertise into the loop, thus strengthening machine learning interactively.
The answer to the second question is to embed the high-dimensional image dataset into a 2D space, and use visualization techniques to enable human experts to validate the generated datasets, correct wrong labels, and enhance the quality of the datasets.
When talking about remote sensing datasets, we often face big datasets, such as BigEarthNet, so the optimization for visualizing big datasets will also be discussed.
We present an image annotation tool; it is inherently implemented with the active learning technique which allows users to interact with it.
As input, the user’s prior knowledge is required, so it will be advantageous when the users are domain experts or trained users with some experiences. More details of this tool can be found in referenced paper [1].
With this image annotation tool, one can achieve on average classification accuracy of more than 90 percent (still to be improved).
In order to observe patterns, relationships and outliers within a dataset, and to validate and clean the generated datasets, a visualization and cleaning tool has been developed, too. It provides interactive visualization, equipped with real-time rendering. The bottleneck in the visualization and cleaning tool is real-time rendering that requires lots of resources, as well as the handling of labels with uncertainties, for instance, points on the border.
Our proposed optimization is to make statistical neighborhood analyses, and a neighborhood decision tree was designed to distinguish four kinds of neighborhood patterns. We consider:
• An outlier is a point which is far away from the main body of data points, and with few neighbors.
• A border case is a point with a varying number of identical or different labels.
• A heterogeneous point is one surrounded by many aliens.
• A homogeneous point is one surrounded by its own groups.
(Illustration)
As shown in the attached figure, two metrics are applied when we discriminate these four groups. On the right upper corner is the embedding space showing the raw dataset, in the bottom and on the left side are the corresponding embedding spaces for each pattern. In the homogeneous case, we notice very smooth pattern changes among the labeled image patches. The other three patterns are “difficult” cases that need further processing.
With this solution, using the proposed visualization and cleaning tool, users can perform visual data mining. We see the low-dimensional embedded feature space, where similar patterns are grouped together. When the mouse is moved over a point in the feature space, the corresponding image is shown as a snapshot image, and the original image appears on the top of left sidebar. When we left-click the mouse, the neighbors of the chosen image are also shown on the bottom of the left sidebar, which provides a hint whether the label is similar to its neighbor or different from its neighbor.
When we go across the feature space, we see the change of patterns within the same class. If we go to a specific label which stands out from its neighbors, a right-click of the mouse generates a context menu, which lets the user assign a new label. When all validations are done, the menu offers the option to generate all updates as a csv file.
Then, with human interaction and neighborhood pattern analysis (afterwards some interactive visual data checks), new qualified datasets can be generated.
Thanks to the active learning techniques, the image annotation tool can increase the efficiency of generating EO image benchmark datasets - due to the facts that it’s fast in speed, robust in processing different types of products, and smart to train with user interactions.
Furthermore, the visualization and cleaning tool enables users to visualize a dataset, and perform an exploration of unknown patterns or classes, and conduct visual data mining.

Reference:
[1] An active learning tool for the generation of Earth observation image benchmarks, Wei Yao, Corneliu Octavian Dumitru, Mihai Datcu, IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Brussels, Belgium, July 11-16, 2021.

Earth observations (EO) provide consistent data over different spatio-temporal scales that can be used for modeling the Earth and its changing environment. Such modeling efforts require characterization of complex and nonlinear processes that could benefit from data-driven techniques - in particular, machine learning (ML). ML techniques are rapidly being applied to EO data to serve numerous and diverse markets, from agriculture to medicine to transportation. These technologies present a game-changing opportunity: the ability to identify and address unique, complex, and emerging challenges at local, regional, and global scales more accurately and more quickly. From poverty alleviation, food security, and climate change adaptation to sustainable resource management and humanitarian response, the combination of EO with ML can help humanity see, understand, and respond to a rapidly changing world.

ML models are built through an iterative process and learn patterns from data. As a result, any uncertainty or bias in the data propagates to the model and future estimates. This highlights the importance of reproducible and reusable data and modeling pipelines to validate the results of ML applications or improve their performance to meet practical requirements. To enable successful applications of ML techniques in Earth science, it’s paramount to build an ecosystem for sharing and publishing benchmark training data, models and best practices. Such an ecosystem can make these techniques more accessible and bring transparency and trust in the results.

In this presentation, I will introduce Radiant MLHub - the first cloud-based open-access repository for geospatial training data and models. Radiant MLHub is an ecosystem that provides access to benchmark training datasets for applications in Earth Sciences, has a community of practices who advance best practices and guidelines for these techniques, and builds integrations with other platforms to advance user experience.

Radiant MLHub is built on open-source community standards, and anyone can access the data for various applications from agricultural monitoring and land cover mapping to infrastructure detection and flood mapping. To harmonize data structures and metadata, Radiant MLHub uses SpatioTemporal Asset Catalog (STAC) specification for its catalog records and stores data in cloud-native formats such as Cloud Optimized GeoTIFF (COG) and GeoJSON.

Radiant MLHub Python Client makes it seamless for users to discover training datasets (and models) using search criteria such as geographical bounding box, temporal range, key words, and Digital Object Identifier (DOI) among others. Users have the option to download the data from the web interface as well.