CMEMS Importer Service

From Gcube Wiki
Jump to: navigation, search

Overview

Usage

REST API

Discover the service

The service is registered in the Information System with the following coordinates:

Group: DataAnalysis

Name: cmems-importer-service

Browsing datasets

List Products

Request: 

GET /api/cmems/products

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products'

Sample output: 

<cmemsProducts>
  <product>
    <abstract>...</abstract>
    <allKeywords>
      numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
    </allKeywords>
    <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
    <creationDate>2016-10-14T00:00:00Z</creationDate>
    ...
  </product>
  <product>
    ...
  </product>
</cmemsProducts>

Get a Single Product

Request: 

GET /api/cmems/products/{productName}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024'

Sample output: 

<product>
  <abstract>...</abstract>
  <allKeywords>
    numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
  </allKeywords>
  <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
  <creationDate>2016-10-14T00:00:00Z</creationDate>
  ...
</product>

List Datasets available under the Subsetter protocol

Request: 

GET /api/cmems/products/{productName}/datasets/tds

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds'

Sample output: 

  <cmemsDatasets>
    <dataset>
      <name>global-analysis-forecast-phy-001-024</name>
      <protocol>TDS</protocol>
      <url>
        http://nrtcmems.mercator-ocean.fr/motu-web/Motu?action=describeproduct&service=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS&product=global-analysis-forecast-phy-001-024
      </url>
    </dataset>
    <dataset>
      ...
    <dataset>
  </cmemsDatasets>

Get Dataset details from the Motu server

Request: 

GET /api/cmems/products/{product}/datasets/tds/{dataset}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds/global-analysis-forecast-phy-001-024'

Sample output: 

  <productMetadataInfo code="0" id="global-analysis-forecast-phy-001-024" lastUpdate="Not Available" msg="OK" title="global-analysis-forecast-phy-001-024" url="">
    <dataGeospatialCoverage>
      <axis axisType="Time" code="0" description="Time (hours since 1950-01-01)" lower="499548" msg="OK" name="time" units="hours since 1950-01-01 00:00:00" upper="596820"/>
      <axis axisType="Lat" code="0" description="Latitude" lower="-80" msg="OK" name="latitude" units="degrees_north" upper="90"/>
      <axis axisType="Lon" code="0" description="Longitude" lower="-180" msg="OK" name="longitude" units="degrees_east" upper="179.9166717529296875"/>
      <axis axisType="Height" code="0" description="Depth" lower="0.4940249919891357421875" msg="OK" name="depth" units="m" upper="5727.9169921875"/>
    </dataGeospatialCoverage>
    <availableDepths>
      0.49402;1.54138;2.64567;3.81949;5.07822;...
    </availableDepths>
    <availableTimes>
      2006-12-27 12:00:00;2006-12-28 12:00:00;2006-12-29 12:00:00;...
    </availableTimes>
    ...
  </productMetadataInfo>

List Motu servers

To retrieve the list of all Motu servers publishing CMEMS data.

Request: 

GET /api/cmems/motu

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/motu'

Sample output: 

  <motuServers>
     <motuServer>
        <endpoint>
          http://cmems-bs-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     <motuServer>
        <endpoint>
          http://cmems-med-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     ...
  </motuServers>

List Geographical Regions

To retrieve the list of Geographical Regions covered by CMEMS data.

Request: 

GET /api/cmems/regions

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/regions'

Sample output: 

  <cmemsRegions>
    <cmemsRegion>
      <name>
        arctic-ocean
      </name>
    </cmemsRegion>
    <cmemsRegion>
      <name>
        baltic-sea
      </name>
    </cmemsRegion>
    ...
  </cmemsRegions>

List Variables

To retrieve the list of Variables present in CMEMS datasets.

Request: 

GET /api/cmems/variables

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/variables'

Sample output: 

  <cmemsVariables>
    <cmemsVariable>
      <name>
        age_of_sea_ice
      </name>
    </cmemsVariable>
    <cmemsVariable>
      <name>
        air_density
      </name>
    </cmemsVariable>
    ...
  </cmemsVariables>

Managing import tasks

Calculate the size of the imported dataset

To estimate the size, in bytes, of the imported dataset.

Request: 

GET /api/tasks/preview/sizes/dataset

URL parameters:

  • p: (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
  • d: (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
  • m: (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
  • v: (optional) the name of variables to import. Defaults to all variables in the dataset. Multiple parameters are allowed here.
  • tlo: (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
  • thi: (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
  • xlo: (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
  • xhi: (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
  • ylo: (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
  • yhi: (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
  • zlo: (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
  • zhi: (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024

Sample output: 

14451431889375

Calculate the size of a chunk of the imported dataset

Request: 

GET /api/tasks/preview/sizes/chunk

URL parameters:

  • All the parameters above, plus the following one:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&s=month

Sample output: 

  107206468022

Schedule an Import task

Request: 

POST /api/tasks

URL parameters: None

Data Params TODO

With curl: TODO

The operation returns the id of the scheduled task. Sample output: 

 ...

Browsing import tasks and reports

Get/search tasks

Return a list tasks, ordered by last execution time (or submit time, if not executed yet). Tasks can be filtered by user, token and scope, provided as query parameters.

Request: 

GET /api/tasks

URL parameters:

  • user: (optional) the name of the user who submitted the task.
  • token: (optional) the token used when submitting the task.
  • scope: (optional) the scope where the task was submitted.
  • limit: (optional) the maximum number of tasks returned.

Sample output: 

  <importTasks>
    <importTask>
      <submissionInfo>
        <scheduled>2018-01-10T19:28:59+02:00</scheduled>
        <scope>/gcube/devNext/NextNext</scope>
        <user>Alice</user>
      </submissionInfo>
      <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
      <lastExecution>
        <begin>2018-01-15T11:41:59+01:00</begin>
        <begin>2018-01-16T32:11:56+01:00</begin>
        <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
        <progress>1.0</progress>
        <reports>
          <report>
            <name>stderr.log</name>
            <size>25</size>
            <snippet>a few error messages here</snippet>
          </report>
          <report>
            <name>stdout.log</name>
            <size>3251</size>
            <snippet>
              Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum
              vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. 
              Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu
              odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
              quis sem elementum, dapib...
            </snippet>
          </report>
        </reports>
        <status>complete</status>
      </lastExecution>
      <importParameters>
        <dataset>datasetname</dataset>
        <motu>http://somehost/...</motu>
        <product>productname</product>
        <variables>
          <variable>salinity</variable>
          <variable>velocity</variable>
        </variables>
        ...
      </importParameters>
    </importTask>
    ...
  </importTask>

Retrieving a task

Return metadata about a single task, including submission details, import parameters, last execution info and reports snippets.

Request: 

GET /api/tasks/{taskId}

No parameters are expected in the URL.

Sample output: 

  <importTask>
    <submissionInfo>
      <scheduled>2018-01-10T19:28:59+02:00</scheduled>
      <scope>/gcube/devNext/NextNext</scope>
      <user>Alice</user>
    </submissionInfo>
    <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
    <lastExecution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>last</id>
      <progress>1.0</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...</snippet>
        </report>
      </reports>
      <status>complete</status>
    </lastExecution>
    <importParameters>
      <dataset>datasetname</dataset>
      <motu>http://somehost/...</motu>
      <product>productname</product>
      <variables>
        <variable>salinity</variable>
        <variable>velocity</variable>
      </variables>
      ...
    </importParameters>
  </importTask>

Retrieving task executions

Return a list of executions for a given task.

Request: 

GET /api/tasks/{taskId}/executions

No parameters are expected in the URL.

Sample output: 

  <executions>
    <execution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
      <progress>0.8725477544390946</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>
            Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
            Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
            viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
            lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
            efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
            tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
          </snippet>
        </report>
      </reports>
      <status>running</status>
    </execution>
    <execution>
      ...
    </execution>
  </executions>

Retrieving a task execution

Return a single execution for a given task.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}

No parameters are expected in the URL.

Sample output: 

  <execution>
    <begin>2018-01-15T11:41:59+01:00</begin>
    <begin>2018-01-16T32:11:56+01:00</begin>
    <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
    <progress>0.8725477544390946</progress>
    <reports>
      <report>
        <name>stderr.log</name>
        <size>25</size>
        <snippet>a few error messages here</snippet>
      </report>
      <report>
        <name>stdout.log</name>
        <size>3251</size>
        <snippet>
          Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
          Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
          viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
          lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
          efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
          tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
        </snippet>
      </report>
    </reports>
    <status>running</status>
  </execution>

Retrieving execution reports

Return a list of execution reports (i.e. logs). For each report, a snippet of ~1000 characters is included.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports

No parameters are expected in the URL.

Sample output: 

  <executionReports>
    <executionReport>
      <name>stderr.log</name>
      <size>25</size>
      <snippet>a few error messages here</snippet>
    </executionReport>
    <executionReport>
      <name>stdout.log</name>
      <size>3251</size>
      <snippet>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
        quis sem elementum, dapib...
      </snippet>
    </executionReport>
  </executionReports>

Retrieving a single report

Retrieve the specified report.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}

No query parameters are expected in the URL.

Sample output: 

  <executionReports>
    <executionReport>
      <name>stderr.log</name>
      <size>25</size>
      <snippet>a few error messages here</snippet>
    </executionReport>
    <executionReport>
      <name>stdout.log</name>
      <size>3251</size>
      <snippet>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
        quis sem elementum, dapib...
      </snippet>
    </executionReport>
  </executionReports>

Retrieving report text only

Retrieve only the text contained in the specified report.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}/text

No query parameters are expected in the URL.

Sample output: 

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
quis sem elementum, dapib...

Updating executions and reports

This interface is meant to be used by the cmems-importer-se-plugin, while running the import task.

Creating a new execution

Create a new execution for an existing import task.

URL

  • /api/tasks/{taskId}/executions

Method

  • POST

URL Params

  • Required:
    • taskId=[String]

Data Params

The body contains execution metadata including:

  • the execution id (String)
  • the current status (String)
  • execution progress (Double within [0,1])
  • the execution start time
  • the execution end time
<execution>
    <id>1517927891274</id>
    <begin>2018-02-06T15:38:13.579+01:00</begin>
    <progress>0.1</progress>
    <status>running</status>
</execution>

Updating an execution

Update an existing execution.

URL

  • /api/tasks/{taskId}/executions/{executionId}

Method

  • PUT

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]

Data Params

The body contains execution metadata including:

  • the current status (String)
  • execution progress (Double within [0,1])
  • the execution end time

Other execution fields (e.g. begin, id, lastUpdate) are ignored. 

<execution>
    <progress>0.5</progress>
    <status>running</status>
</execution>

Adding a report

Create a new report for an existing task execution.

URL

  • /api/tasks/{taskId}/executions/{executionId}/reports

Method

  • POST

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]

Data Params

The body contains report metadata including:

  • the report name (String)
  • the report text (String)
<executionReport>
  <name>out.log</name>
  <text>report text...</text>
</executionReport>

Updating a report

Update an existing report.

URL

  • /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}

Method

  • PUT

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]
    • reportName=[String]

Data Params

The body must contain the updated text of the report.

  • the report text (String)
<executionReport>
  <text>udpated report text...</text>
</executionReport>
Retrieved from "https://wiki.gcube-system.org/index.php?title=CMEMS_Importer_Service&oldid=31014"