Tutorial for Sentinel-3 OLCI data

This tutorial demonstrates step by step how to use felyx with Sentinel-3 ocean colour data from OLCI sensor.

felyx installation

In this tutorial, we will use the felyx demo version provided on a virtualbox image at:

The installation procedure for the virtualbox image is fully described in felyx online documentation at:

In the following, we assume the virtualbox has been successfully installed and started. We also assumed you have also correctly setup the access to the shared data folders, as explained in the documentation.

felyx configuration

In this section, we will configure felyx to use OLCI products.

If the previous step went smoothly, you should be able to open the felyx web interface from your host machine (the machine on which your virtualbox is run) at the following address:

This should display the following screen:

_images/welcome.png

Go into the configuration section (configure item in the top menu):

_images/configuration.png

This page presents the different configuration steps in felyx.

We will first configure the dataset we want to extract miniprods and metrics from. Click on the Dataset configuration button on the left. This open the catalogue view, which for now is empty since we did not configure any dataset yet:

_images/datasets.png

Click on the Add a dataset button and open the configuration page for a new dataset. This opens a form that you will have to fill in for each new dataset.

In this tutorial, we will use the full resolution OLCI Water product (OL_2_WFR). You have to fill in the following fields:

  • Name : a short identifier for the dataset (here S3A_OL_2_WFR), you can invent your own code here but use preferably an existing code for the product (if any) to avoid confusion later.
  • Description, Originator, Supplier, Level and Version are metadata fields used only for dataset discovery. You can put whatever you want here.
  • Data model is the observation pattern corresponding to the dataset, among the patterns proposed in cerbere library. You have to refer to cerbere documentation to correctly choose the data model, or felyx won’t work with your dataset. In our case the pattern for OL_2_WFR is Swath.
  • Data mapper is the reader corresponding to the format of your dataset, among the readers provided in cerbere library. Again you have to refer to cerbere documentation here and possibly write your own mapper class if it is not yet handled by cerbere. Luckily, in our case, there is already a mapper for OLCI products (SAFEOLFile)
  • Keywords is also a metadata field for discovery where we recommend to use GCMD keywords (click on the link). We choose here: Oceans > Ocean Colour
  • Tags is a free list of comma separated words than can be used later as selection filter. For instance here: olci, sentinel-3, ocean-colour

Warning

In above fields, don’t use characters such as ‘.’ or ‘/’.

Here is a screenshot of a proper configuration for this product:

_images/datasetconf.png

Click on Save button when the form is completed. You are now back to the catalogue view with one dataset in the list:

_images/datasets2.png

The next step is to configure the extraction sites for the miniprod. It is possible to define the sites and site collections from the web interface (Site configuration) but it is also much easier and faster to start from a predefined list of sites that we will import to the system.

The format and import mechanism are described in the felyx documentation here. http://felyx.readthedocs.org/en/latest/configuring_sites.html

In this tutorial, we will use the list of sites defined for GHRSST project, which can be downloaded here:

Note

alternatively, you can use also other collection files, available in the same repository, such as ‘aeronet.csv’.

Download this file, open a terminal and import the sites following the method described in the documentation. The URL for the import service is the felyx root URL (‘http://localhost:7319‘) followed by the ‘/services/csv/’ path:

curl -XPOST -F type=static -F file=@ghrsst.csv 'http://localhost:7319/services/csv/'

If everything goes fine, you should have the following message in your terminal:

{"status": "ok", "msg": "Upload complete.\nDon't forget to use felyx-cache to update backend database"}

You can check the result of this import by going back to the configuration web interface, and clicking on Sites configuration button. The interface now displays the list of sites configured. Click on a specific site in the list, the map view will zoom in on this site and show its shape and description:

_images/siteview.png

It is possible to display the list of sites belonging to a given collection, by clicking on Collections in the search bar and clicking again on a given collection (here GHRSST).

_images/collectionview.png

We now want to extract miniprods from S3A_OL_2_WFR products over the sites defined in GHRSST site collection. We need to associate this site collection with the dataset. This is done again in the configuration section of the web interface, by clicking on Extraction configuration, which opens the list of miniprods extractions already configured in felyx. For now, it is empty and looks like this:

_images/extractions.png

Let’s click on Add an extraction to configure our first extraction:

_images/extractionconf.png

You must select here one (or more) datasets and one (or more) sites collections over which felyx will extract the miniprods (and later metrics) from the selected datasets. Here it is easy because we only have one choice in each category.

We will ignore for now the other options which are meant for dynamic site collections only. Therefore we can click on Save and come back to the main extraction list which is now updated:

_images/extractions2.png

Warning

Metrics will be defined per extraction. If you intend to define different metrics for different site collections, you should create two different extractions, one for each collection.

We can now produce some miniprods!!

Miniprod generation

Now we go back into the virtual machine terminal:

_images/vm0.png

Note

If you have properly configured the ssh port redirection (during the VM installation), you can also directly connect to the VM from your local terminal. It is much more convenient!

ssh -p 2000 felyx@localhost

First, to access the felyx backend commands, you will have to load the virtual environment. In a terminal on your virtual machine, enter:

source worker/bin/activate

We then have to clean and repopulate the felyx cache which stores a copy of the configuration we just entered through the web interface:

felyx-cache
_images/felyxcache.png

If you followed the installation instructions, the shared folder for the data should be accessible under:

/data/felyx

In the installation documentation, this corresponds to the following repository on the host (but it will be different in your case):

/home1/ananda/felyx/data

We will create first some folders here to organize our data production:

# folder for the input data from which to extract miniprods
mkdir /home1/ananda/felyx/data/test_input_data
# folder for created internal outputs
mkdir /home1/ananda/felyx/data/workspace/
# manifest files from miniprod production
mkdir /home1/ananda/felyx/data/workspace/miniprod-manifests
# miniprod archive
mkdir /home1/ananda/felyx/data/miniprods

Now let’s copy some OLCI WFR Level 2 data in the shared folder of the virtual machine:

cp /home1/ananda/data/sentinel-3/olci/S3A_OL_2_WFR/* /home1/ananda/felyx/data/test_input_data

We will now extract the miniprods over an input data file we just copied. This is done through the felyx-miniprod command, using the following generic form:

felyx-miniprod <input OLCI or SLSTR file> <OLCI or SLSTR dataset code> --mode local --no-dynamic -o <output for manifest files>

For example, type in the virtual machine terminal:

felyx-miniprod /data/felyx/test_input_data/S3A_OL_2_WFR/S3A_OL_2_WFR____20080108T065652_20080108T074025_20150521T130759_2613_084_049______MAR_D_NT_001.SEN3 S3A_OL_2_WFR --mode local --no-dynamic -o /data/felyx/workspace/miniprod-manifests

For more explanation on the options, refer to the felyx-miniprod command documentation.

The --no-dynamic arguments is used here to skip the production of metrics for dynamic miniprods.

Running on one single file in local mode (--mode=local) allows to quickly assess if something is going wrong. In a standalone virtual machine, you don’t have to use any other mode as no distributed processing is possible.

A manifest file is created in -o repository for each input Sentinel-3 file passed to felyx-miniprod command : it contains the path to each miniprod that has been extracted from this input file.

If it runned smoothy, we can now process all available input files in the shared folder:

find /data/felyx/test_input_data/S3A_OL_2_WFR/ -name "S3A_OL_2_WFR*.nc" | xargs -ifoo felyx-miniprod foo --mode local --no-dynamic -o /data/felyx/workspace/miniprod-manifests

If there were input files matching some extraction sites, some miniprods should have been extracted in the miniprod storage area of the shared disk:

/data/felyx/workspace/

Note

If there aren’t any miniprod files produced, it can just mean the input files you used were not intersecting any of the extraction sites defined in the site collections.

Metrics production

Configure

We will now process some metrics from these miniprods. First go to the metrics configuration page, by clicking on metrics configuration in the main configuration page. We have for now an empty list of metrics.

_images/metrics.png

Click on Add a metric button to define a new metric. We are presented with a form where we have to define the following fields:

  • a identifier: a code name for the metric which will be used to query this metric from the felyx API.
  • a descriptive label: this will be used in the web interface to provide a user friendly name for this metric.
  • the attached extraction: this designates the miniprods from which we will compute this metric. In this example, we defined one extraction only so only one choice is presented in this list.
  • an operation: this is the calculation defined to produce the metric. By clicking on Add an operator you open a new form where you can define this operation.

The operation definition form presents a list of available operators you can use (like in Excel):

_images/operator1.png

Click on the mean operator, this opens again a form where you will detail the arguments to this operator:

  • the field on which to apply our operator. Here we choose the field Oa01_reflectance.
  • limit_to_site impose to use only the pixels included within the limits of the static (or dynamic) site. A miniprod usually covers a complete or a section of site, some pixels being within or out of the boundaries of this site. Here by leaving this field blank (or entering True), we only consider the pixels with the site limits for the mean value estimation.
  • must_have is where you would define additional constraints on the pixel selection before the mean value calculation. You could here for instance decide to select a pixel for which a quality flag (also in the extracted miniprods) is greater than a certain value. We leave it blank in this example.
  • axis: don’t use this.

The resulting form should look like this:

_images/operator2.png

Yo can now click on Add operator. You are back on the main operation form, where you can click on save button:

_images/operator3.png

We are now back on to the list of defined metrics, with a new metric being added to this list:

_images/metrics2.png

Generate

We are now ready to produce some metrics. Like for the miniprod production, this is done through some command line tools.

First activate the felyx-cache command, since we modified the configuration (or no metrics would be produced).

felyx-cache

When we created the miniprods, we also generated some manifest files containg the list of miniprods created for each input file. We can use these files as inputs to the metric generation tool. They should be located under:

ls /data/felyx/workspace/miniprod-manifests/S3A_OL_2_WFR/

We will now create an output folder for the metric files:

mkdir /data/felyx/workspace/metrics/S3A_OL_2_WFR/

Let’s generate some metrics now:

find /data/felyx/workspace/miniprod-manifests/S3A_OL_2_WFR/ -name "*.manifest" | xargs -ifoo \
felyx-metric -o /data/felyx/workspace/metrics/S3A_OL_2_WFR --mode local --no-clean --storage \
file foo

What we just did here is to ask the metrics generation for all the miniprods listed in the manifest files (in data/felyx/workspace/miniprod-manifests/S3A_OL_2_WFR/)

We specified the identifier of the dataset (S3A_OL_2_WFR) so that the system can query the proper configuration and know which metric it should compute for these particular files. The system search which miniprod extractions are associated to this dataset, and which metric it should process for a given miniprod at a given site.

Using --no-clean keyword keeps the manifest files instead of deleting them after the metrics are created.

Using --storage file creates the metrics as json files (alternative is to directly register them into felyx).

We also specified the output folder for the metric files (in json format) with -o option.

We can go to this output folder to check if there were any metrics created. We should have metric files for both the miniprod metadata (created by default) and the mean_Oa01_reflectance metric we have configured previously.

We have now to register these metrics into felyx (in the ElasticSearch index). This is done through a third command line tool, felyx-metric-index:

find /data/felyx/workspace/metrics/S3A_OL_2_WFR/ -name "*.json" | felyx-metric-index --stdin

The metrics and miniprods metadata are now registered in felyx can be queried.

Sentinel-3 SLSTR data

Let’s play now with SLSTR data.

This case is more complex that with OLCI products because of the different sub-products (with different grids and views) existing within a single SAFE container. Because these sub-products don’t all overlap (e.g. they don’t have the same array dimensions nor the same pixel locations), we can not extract a single miniprod from a SAFE container.

The data in felyx are read with cerbere python package. The cerbere strategy for SLSTR products is to provide several mappers adapted to each product/subproduct combination. A mapper will merge the nadir and oblique view fields as if they were in the same product file.

This means that in felyx you have to configure a dataset for each possible combination of a SLSTR product and cerbere mapper. The following table tells you which mapper to use to read which data:

Product or Sub-product Mapper class Datamodel
S3A_SL_2_WCT Nadir (in) SAFESLIRFile Swath
S3A_SL_1_RBT 1km Nadir (in)
S3A_SL_2_WCT Oblique (io)
S3A_SL_1_RBT 1km Oblique (io)
S3A_SL_1_RBT 500m & SWIR A Stripe Nadir (an) SAFESL500AFile Swath
S3A_SL_1_RBT 500m & SWIR A Stripe Oblique (ao)
S3A_SL_1_RBT 500m & SWIR B Stripe Nadir (bn) SAFESL500BFile Swath
S3A_SL_1_RBT 500m & SWIR B Stripe Oblique (bo)
S3A_SL_1_RBT TDI Nadir (cn) SAFESL500TDIFile Swath
S3A_SL_1_RBT TDI Oblique (co)

There are 4 different possible mappers for a S3A_SL_1_RBT product which means you will have to configure 4 datasets in felyx if you want to extract miniprod for all possible sub-products. This is the more tricky case. Fortunately there is only one possible mapper for any other S3 SLSTR product!

Following the method described before, we will configure 4 datasets to cover all possible subproducts in a EUMETSAT S3A_SL_1_RBT product.

For S3A_SL_2_WCT, only one dataset needs to be created, with the mapper SAFESLIRFile associated to it.

We then configure extractions, in the same way we did for the OLCI product.

Here we go again, creating for instance miniprods for our S3A_SL_2_WCT sample datasets:

find /data/felyx/test_input_data/S3A_SL_2_WCT/ -name "S3A_SL_2_WCT_*.SEN3" | \
xargs -ifoo felyx-miniprod --no-dynamic --mode local foo S3A_SL_2_WCT \
-o /data/felyx/workspace/miniprod-manifests/

We then configure a metric for this dataset, which will be the mean of N3_sea_surface_temperature (code mean_sst), using the configuration interface, as demonstrated for OLCI.

Refresh the configuration cache:

felyx-cache

Then run the production of the metrics:

find /data/felyx/workspace/miniprod-manifests/ -name "*.manifest" | xargs -ifoo \
felyx-metric -o /data/felyx/workspace/metrics/S3A_SL_2_WCT --mode local --no-clean \
--storage file foo

And register them:

find /data/felyx/workspace/metrics/S3A_SL_2_WCT/ -name "*.json" | felyx-metric-index --stdin