Difference between revisions of "CMEMS Importer Service"

From Gcube Wiki
Jump to: navigation, search
(→‎Updating executions and reports =)
(→‎Creating a new execution)
Line 558: Line 558:
 
=== Updating executions and reports ===
 
=== Updating executions and reports ===
  
−
==== Creating a new execution ====  
+
==== Creating a new execution ====
 +
 
 +
Create a new execution for an existing import task.
 +
 
 +
'''URL'''
 +
 
 +
*/api/tasks/{taskId}/executions
 +
 
 +
'''Method'''
 +
 
 +
*POST
 +
 
 +
'''URL Params'''
 +
* Required:
 +
** taskId=[String]
 +
 
 +
'''Data Params'''
  
 
==== Updating an execution ====
 
==== Updating an execution ====

Revision as of 16:56, 6 February 2018

Overview

Usage

REST API

Discover the service

The service is registered in the Information System with the following coordinates:

Group: DataAnalysis

Name: cmems-importer-service

Browsing datasets

List Products

Request: 

GET /api/cmems/products

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products'

Sample output: 

<cmemsProducts>
  <product>
    <abstract>...</abstract>
    <allKeywords>
      numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
    </allKeywords>
    <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
    <creationDate>2016-10-14T00:00:00Z</creationDate>
    ...
  </product>
  <product>
    ...
  </product>
</cmemsProducts>

Get a Single Product

Request: 

GET /api/cmems/products/{productName}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024'

Sample output: 

<product>
  <abstract>...</abstract>
  <allKeywords>
    numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
  </allKeywords>
  <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
  <creationDate>2016-10-14T00:00:00Z</creationDate>
  ...
</product>

List Datasets available under the Subsetter protocol

Request: 

GET /api/cmems/products/{productName}/datasets/tds

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds'

Sample output: 

  <cmemsDatasets>
    <dataset>
      <name>global-analysis-forecast-phy-001-024</name>
      <protocol>TDS</protocol>
      <url>
        http://nrtcmems.mercator-ocean.fr/motu-web/Motu?action=describeproduct&service=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS&product=global-analysis-forecast-phy-001-024
      </url>
    </dataset>
    <dataset>
      ...
    <dataset>
  </cmemsDatasets>

Get Dataset details from the Motu server

Request: 

GET /api/cmems/products/{product}/datasets/tds/{dataset}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds/global-analysis-forecast-phy-001-024'

Sample output: 

  <productMetadataInfo code="0" id="global-analysis-forecast-phy-001-024" lastUpdate="Not Available" msg="OK" title="global-analysis-forecast-phy-001-024" url="">
    <dataGeospatialCoverage>
      <axis axisType="Time" code="0" description="Time (hours since 1950-01-01)" lower="499548" msg="OK" name="time" units="hours since 1950-01-01 00:00:00" upper="596820"/>
      <axis axisType="Lat" code="0" description="Latitude" lower="-80" msg="OK" name="latitude" units="degrees_north" upper="90"/>
      <axis axisType="Lon" code="0" description="Longitude" lower="-180" msg="OK" name="longitude" units="degrees_east" upper="179.9166717529296875"/>
      <axis axisType="Height" code="0" description="Depth" lower="0.4940249919891357421875" msg="OK" name="depth" units="m" upper="5727.9169921875"/>
    </dataGeospatialCoverage>
    <availableDepths>
      0.49402;1.54138;2.64567;3.81949;5.07822;...
    </availableDepths>
    <availableTimes>
      2006-12-27 12:00:00;2006-12-28 12:00:00;2006-12-29 12:00:00;...
    </availableTimes>
    ...
  </productMetadataInfo>

List Motu servers

To retrieve the list of all Motu servers publishing CMEMS data.

Request: 

GET /api/cmems/motu

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/motu'

Sample output: 

  <motuServers>
     <motuServer>
        <endpoint>
          http://cmems-bs-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     <motuServer>
        <endpoint>
          http://cmems-med-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     ...
  </motuServers>

List Geographical Regions

To retrieve the list of Geographical Regions covered by CMEMS data.

Request: 

GET /api/cmems/regions

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/regions'

Sample output: 

  <cmemsRegions>
    <cmemsRegion>
      <name>
        arctic-ocean
      </name>
    </cmemsRegion>
    <cmemsRegion>
      <name>
        baltic-sea
      </name>
    </cmemsRegion>
    ...
  </cmemsRegions>

List Variables

To retrieve the list of Variables present in CMEMS datasets.

Request: 

GET /api/cmems/variables

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/variables'

Sample output: 

  <cmemsVariables>
    <cmemsVariable>
      <name>
        age_of_sea_ice
      </name>
    </cmemsVariable>
    <cmemsVariable>
      <name>
        air_density
      </name>
    </cmemsVariable>
    ...
  </cmemsVariables>

Managing import tasks

Calculate the size of the imported dataset

To estimate the size, in bytes, of the imported dataset.

Request: 

GET /api/tasks/preview/sizes/dataset

URL parameters:

  • p: (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
  • d: (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
  • m: (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
  • v: (optional) the name of variables to import. Defaults to all variables in the dataset. Multiple parameters are allowed here.
  • tlo: (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
  • thi: (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
  • xlo: (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
  • xhi: (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
  • ylo: (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
  • yhi: (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
  • zlo: (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
  • zhi: (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024

Sample output: 

14451431889375

Calculate the size of a chunk of the imported dataset

Request: 

GET /api/tasks/preview/sizes/chunk

URL parameters:

  • All the parameters above, plus the following one:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&s=month

Sample output: 

  107206468022

Schedule an Import task

Request: 

GET /api/tasks/schedule

URL parameters:

  • All the parameters above, plus the following ones:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.
  • b: (optional) how many days in the past data should not be imported, wrt the import execution time. If not set, defaults to 0.
  • f: (mandatory) a cron-compliant expression to set how often the import task should run. The importer will not execute more than import task per day, per scheduled task.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/schedule?p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&s=month

The operation returns the id of the scheduled task. Sample output: 

 ...

Browsing import tasks and reports

Get/search tasks

Return a list tasks, ordered by last execution time (or submit time, if not executed yet). Tasks can be filtered by user, token and scope, provided as query parameters.

Request: 

GET /api/tasks

URL parameters:

  • user: (optional) the name of the user who submitted the task.
  • token: (optional) the token used when submitting the task.
  • scope: (optional) the scope where the task was submitted.
  • limit: (optional) the maximum number of tasks returned.

Sample output: 

  <importTasks>
    <importTask>
      <submissionInfo>
        <scheduled>2018-01-10T19:28:59+02:00</scheduled>
        <scope>/gcube/devNext/NextNext</scope>
        <user>Alice</user>
      </submissionInfo>
      <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
      <lastExecution>
        <begin>2018-01-15T11:41:59+01:00</begin>
        <begin>2018-01-16T32:11:56+01:00</begin>
        <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
        <progress>1.0</progress>
        <reports>
          <report>
            <name>stderr.log</name>
            <size>25</size>
            <snippet>a few error messages here</snippet>
          </report>
          <report>
            <name>stdout.log</name>
            <size>3251</size>
            <snippet>
              Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum
              vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. 
              Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu
              odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
              quis sem elementum, dapib...
            </snippet>
          </report>
        </reports>
        <status>complete</status>
      </lastExecution>
      <importParameters>
        <dataset>datasetname</dataset>
        <motu>http://somehost/...</motu>
        <product>productname</product>
        <variables>
          <variable>salinity</variable>
          <variable>velocity</variable>
        </variables>
        ...
      </importParameters>
    </importTask>
    ...
  </importTask>

Retrieving a task

Return metadata about a single task, including submission details, import parameters, last execution info and reports snippets.

Request: 

GET /api/tasks/{taskId}

No parameters are expected in the URL.

Sample output: 

  <importTask>
    <submissionInfo>
      <scheduled>2018-01-10T19:28:59+02:00</scheduled>
      <scope>/gcube/devNext/NextNext</scope>
      <user>Alice</user>
    </submissionInfo>
    <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
    <lastExecution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>last</id>
      <progress>1.0</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...</snippet>
        </report>
      </reports>
      <status>complete</status>
    </lastExecution>
    <importParameters>
      <dataset>datasetname</dataset>
      <motu>http://somehost/...</motu>
      <product>productname</product>
      <variables>
        <variable>salinity</variable>
        <variable>velocity</variable>
      </variables>
      ...
    </importParameters>
  </importTask>

Retrieving task executions

Return a list of executions for a given task.

Request: 

GET /api/tasks/{taskId}/executions

No parameters are expected in the URL.

Sample output: 

  <executions>
    <execution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
      <progress>0.8725477544390946</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>
            Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
            Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
            viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
            lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
            efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
            tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
          </snippet>
        </report>
      </reports>
      <status>running</status>
    </execution>
    <execution>
      ...
    </execution>
  </executions>

Retrieving a task execution

Return a single execution for a given task.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}

No parameters are expected in the URL.

Sample output: 

  <execution>
    <begin>2018-01-15T11:41:59+01:00</begin>
    <begin>2018-01-16T32:11:56+01:00</begin>
    <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
    <progress>0.8725477544390946</progress>
    <reports>
      <report>
        <name>stderr.log</name>
        <size>25</size>
        <snippet>a few error messages here</snippet>
      </report>
      <report>
        <name>stdout.log</name>
        <size>3251</size>
        <snippet>
          Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
          Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
          viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
          lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
          efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
          tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
        </snippet>
      </report>
    </reports>
    <status>running</status>
  </execution>

Retrieving execution reports

Return a list of execution reports (i.e. logs). For each report, a snippet of ~1000 characters is included.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports

No parameters are expected in the URL.

Sample output: 

  <executionReports>
    <executionReport>
      <name>stderr.log</name>
      <size>25</size>
      <snippet>a few error messages here</snippet>
    </executionReport>
    <executionReport>
      <name>stdout.log</name>
      <size>3251</size>
      <snippet>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
        quis sem elementum, dapib...
      </snippet>
    </executionReport>
  </executionReports>

Updating executions and reports

Creating a new execution

Create a new execution for an existing import task.

URL

  • /api/tasks/{taskId}/executions

Method

  • POST

URL Params

  • Required:
    • taskId=[String]

Data Params

Updating an execution

Adding a report

Updating a report

Retrieved from "https://wiki.gcube-system.org/index.php?title=CMEMS_Importer_Service&oldid=30853"