Difference between revisions of "CMEMS Importer Service"

From Gcube Wiki
Jump to: navigation, search
(→‎Calculate the size of a chunk of the imported dataset)
(→‎Schedule an Import task)
 
(20 intermediate revisions by 2 users not shown)
Line 13: Line 13:
 
Name: cmems-importer-service
 
Name: cmems-importer-service
  
−
===List Products===
+
===Browsing datasets===
 +
 
 +
====List Products====
  
 
Request:
 
Request:
Line 43: Line 45:
 
</source>
 
</source>
  
−
===Get a Single Product===
+
====Get a Single Product====
  
 
Request:
 
Request:
Line 68: Line 70:
 
</source>
 
</source>
  
−
===List Datasets available under the Subsetter protocol===
+
====List Datasets available under the Subsetter protocol====
  
 
Request:
 
Request:
Line 97: Line 99:
 
</source>
 
</source>
  
−
===Get Dataset details from the Motu server===
+
====Get Dataset details from the Motu server====
  
 
Request:
 
Request:
Line 128: Line 130:
 
</source>
 
</source>
  
−
===List Motu servers===
+
====List Motu servers====
  
 
To retrieve the list of all Motu servers publishing CMEMS data.
 
To retrieve the list of all Motu servers publishing CMEMS data.
Line 159: Line 161:
 
</source>
 
</source>
  
−
===List Geographical Regions===
+
====List Geographical Regions====
  
 
To retrieve the list of Geographical Regions covered by CMEMS data.
 
To retrieve the list of Geographical Regions covered by CMEMS data.
Line 190: Line 192:
 
</source>
 
</source>
  
−
===List Variables===
+
====List Variables====
  
 
To retrieve the list of Variables present in CMEMS datasets.
 
To retrieve the list of Variables present in CMEMS datasets.
Line 221: Line 223:
 
</source>
 
</source>
  
−
===Calculate the size of the imported dataset===
+
===Managing import tasks===
 +
 
 +
====Calculate the size of the imported dataset====
  
 
To estimate the size, in bytes, of the imported dataset.
 
To estimate the size, in bytes, of the imported dataset.
Line 234: Line 238:
 
* '''d''': (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
 
* '''d''': (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
 
* '''m''': (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
 
* '''m''': (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
−
* '''v''': (optional) the name of variables to import. Defaults to all variables in the dataset.
+
* '''v''': (optional) the name of variables to import. Defaults to all variables in the dataset. Multiple parameters are allowed here.
 
* '''tlo''': (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
 
* '''tlo''': (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
 
* '''thi''': (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
 
* '''thi''': (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
Line 259: Line 263:
 
</source>
 
</source>
  
−
===Calculate the size of a chunk of the imported dataset===
+
====Calculate the size of a chunk of the imported dataset====
  
 
Request:
 
Request:
Line 281: Line 285:
 
</source>
 
</source>
  
−
===OP===
+
====Schedule an Import task====
  
 
Request:
 
Request:
 
<source lang="java">
 
<source lang="java">
−
GET /api/
+
POST /api/tasks
 
</source>
 
</source>
  
 
URL parameters:
 
URL parameters:
−
* '''v''': desc here
+
None
 +
 
 +
Data Params
 +
TODO
  
 
With curl:
 
With curl:
−
<source lang="php">
+
TODO
 +
 
 +
The operation returns the id of the scheduled task.
 +
Sample output:
 +
<source lang="xml">
 +
...
 
</source>
 
</source>
 +
 +
===Browsing import tasks and reports===
 +
 +
====Get/search tasks====
 +
 +
Return a list tasks, ordered by last execution time (or submit time, if not executed yet). Tasks can be filtered by user, token and scope, provided as query parameters.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks
 +
</source>
 +
 +
URL parameters:
 +
 +
* '''user''': (optional) the name of the user who submitted the task.
 +
* '''token''': (optional) the token used when submitting the task.
 +
* '''scope''': (optional) the scope where the task was submitted.
 +
* '''limit''': (optional) the maximum number of tasks returned.
  
 
Sample output:
 
Sample output:
 
<source lang="xml">
 
<source lang="xml">
 +
  <importTasks>
 +
    <importTask>
 +
      <submissionInfo>
 +
        <scheduled>2018-01-10T19:28:59+02:00</scheduled>
 +
        <scope>/gcube/devNext/NextNext</scope>
 +
        <user>Alice</user>
 +
      </submissionInfo>
 +
      <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
 +
      <lastExecution>
 +
        <begin>2018-01-15T11:41:59+01:00</begin>
 +
        <begin>2018-01-16T32:11:56+01:00</begin>
 +
        <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
 +
        <progress>1.0</progress>
 +
        <reports>
 +
          <report>
 +
            <name>stderr.log</name>
 +
            <size>25</size>
 +
            <snippet>a few error messages here</snippet>
 +
          </report>
 +
          <report>
 +
            <name>stdout.log</name>
 +
            <size>3251</size>
 +
            <snippet>
 +
              Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum
 +
              vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie.
 +
              Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu
 +
              odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus
 +
              quis sem elementum, dapib...
 +
            </snippet>
 +
          </report>
 +
        </reports>
 +
        <status>complete</status>
 +
      </lastExecution>
 +
      <importParameters>
 +
        <dataset>datasetname</dataset>
 +
        <motu>http://somehost/...</motu>
 +
        <product>productname</product>
 +
        <variables>
 +
          <variable>salinity</variable>
 +
          <variable>velocity</variable>
 +
        </variables>
 +
        ...
 +
      </importParameters>
 +
    </importTask>
 +
    ...
 +
  </importTask>
 
</source>
 
</source>
 +
 +
====Retrieving a task====
 +
 +
Return metadata about a single task, including submission details, import parameters, last execution info and reports snippets.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}
 +
</source>
 +
 +
No parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
  <importTask>
 +
    <submissionInfo>
 +
      <scheduled>2018-01-10T19:28:59+02:00</scheduled>
 +
      <scope>/gcube/devNext/NextNext</scope>
 +
      <user>Alice</user>
 +
    </submissionInfo>
 +
    <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
 +
    <lastExecution>
 +
      <begin>2018-01-15T11:41:59+01:00</begin>
 +
      <begin>2018-01-16T32:11:56+01:00</begin>
 +
      <id>last</id>
 +
      <progress>1.0</progress>
 +
      <reports>
 +
        <report>
 +
          <name>stderr.log</name>
 +
          <size>25</size>
 +
          <snippet>a few error messages here</snippet>
 +
        </report>
 +
        <report>
 +
          <name>stdout.log</name>
 +
          <size>3251</size>
 +
          <snippet>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...</snippet>
 +
        </report>
 +
      </reports>
 +
      <status>complete</status>
 +
    </lastExecution>
 +
    <importParameters>
 +
      <dataset>datasetname</dataset>
 +
      <motu>http://somehost/...</motu>
 +
      <product>productname</product>
 +
      <variables>
 +
        <variable>salinity</variable>
 +
        <variable>velocity</variable>
 +
      </variables>
 +
      ...
 +
    </importParameters>
 +
  </importTask>
 +
</source>
 +
 +
====Retrieving task executions====
 +
 +
Return a list of executions for a given task.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}/executions
 +
</source>
 +
 +
No parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
 +
  <executions>
 +
    <execution>
 +
      <begin>2018-01-15T11:41:59+01:00</begin>
 +
      <begin>2018-01-16T32:11:56+01:00</begin>
 +
      <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
 +
      <progress>0.8725477544390946</progress>
 +
      <reports>
 +
        <report>
 +
          <name>stderr.log</name>
 +
          <size>25</size>
 +
          <snippet>a few error messages here</snippet>
 +
        </report>
 +
        <report>
 +
          <name>stdout.log</name>
 +
          <size>3251</size>
 +
          <snippet>
 +
            Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien.
 +
            Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis
 +
            viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae 
 +
            lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas
 +
            efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla
 +
            tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
 +
          </snippet>
 +
        </report>
 +
      </reports>
 +
      <status>running</status>
 +
    </execution>
 +
    <execution>
 +
      ...
 +
    </execution>
 +
  </executions>
 +
 +
</source>
 +
 +
====Retrieving a task execution====
 +
 +
Return a single execution for a given task.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}/executions/{executionId}
 +
</source>
 +
 +
No parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
 +
  <execution>
 +
    <begin>2018-01-15T11:41:59+01:00</begin>
 +
    <begin>2018-01-16T32:11:56+01:00</begin>
 +
    <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
 +
    <progress>0.8725477544390946</progress>
 +
    <reports>
 +
      <report>
 +
        <name>stderr.log</name>
 +
        <size>25</size>
 +
        <snippet>a few error messages here</snippet>
 +
      </report>
 +
      <report>
 +
        <name>stdout.log</name>
 +
        <size>3251</size>
 +
        <snippet>
 +
          Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien.
 +
          Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis
 +
          viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae 
 +
          lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas
 +
          efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla
 +
          tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
 +
        </snippet>
 +
      </report>
 +
    </reports>
 +
    <status>running</status>
 +
  </execution>
 +
 +
</source>
 +
 +
====Retrieving execution reports ====
 +
 +
Return a list of execution reports (i.e. logs). For each report, a snippet of ~1000 characters is included.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}/executions/{executionId}/reports
 +
</source>
 +
 +
No parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
 +
  <executionReports>
 +
    <executionReport>
 +
      <name>stderr.log</name>
 +
      <size>25</size>
 +
      <snippet>a few error messages here</snippet>
 +
    </executionReport>
 +
    <executionReport>
 +
      <name>stdout.log</name>
 +
      <size>3251</size>
 +
      <snippet>
 +
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
 +
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
 +
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
 +
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
 +
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus
 +
        quis sem elementum, dapib...
 +
      </snippet>
 +
    </executionReport>
 +
  </executionReports>
 +
 +
</source>
 +
 +
====Retrieving a single report ====
 +
 +
Retrieve the specified report.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}
 +
</source>
 +
 +
No query parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
 +
  <executionReports>
 +
    <executionReport>
 +
      <name>stderr.log</name>
 +
      <size>25</size>
 +
      <snippet>a few error messages here</snippet>
 +
    </executionReport>
 +
    <executionReport>
 +
      <name>stdout.log</name>
 +
      <size>3251</size>
 +
      <snippet>
 +
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
 +
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
 +
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
 +
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
 +
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus
 +
        quis sem elementum, dapib...
 +
      </snippet>
 +
    </executionReport>
 +
  </executionReports>
 +
 +
</source>
 +
 +
====Retrieving report text only====
 +
 +
Retrieve only the text contained in the specified report.
 +
 +
Request:
 +
<source lang="java">
 +
GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}/text
 +
</source>
 +
 +
No query parameters are expected in the URL.
 +
 +
Sample output:
 +
<source lang="xml">
 +
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
 +
consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
 +
vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
 +
suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
 +
lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus
 +
quis sem elementum, dapib...
 +
</source>
 +
 +
=== Updating executions and reports ===
 +
 +
This interface is meant to be used by the cmems-importer-se-plugin, while running the import task.
 +
 +
==== Creating a new execution ====
 +
 +
Create a new execution for an existing import task.
 +
 +
'''URL'''
 +
 +
*/api/tasks/{taskId}/executions
 +
 +
'''Method'''
 +
 +
*POST
 +
 +
'''URL Params'''
 +
* Required:
 +
** taskId=[String]
 +
 +
'''Data Params'''
 +
 +
The body contains execution metadata including:
 +
* the execution id (String)
 +
* the current status (String)
 +
* execution progress (Double within [0,1])
 +
* the execution start time
 +
* the execution end time
 +
 +
<source lang="xml">
 +
<execution>
 +
    <id>1517927891274</id>
 +
    <begin>2018-02-06T15:38:13.579+01:00</begin>
 +
    <progress>0.1</progress>
 +
    <status>running</status>
 +
</execution>
 +
</source>
 +
 +
==== Updating an execution ====
 +
 +
Update an existing execution.
 +
 +
'''URL'''
 +
 +
*/api/tasks/{taskId}/executions/{executionId}
 +
 +
'''Method'''
 +
 +
*PUT
 +
 +
'''URL Params'''
 +
* Required:
 +
** taskId=[String]
 +
** executionId=[String]
 +
 +
'''Data Params'''
 +
 +
The body contains execution metadata including:
 +
* the current status (String)
 +
* execution progress (Double within [0,1])
 +
* the execution end time
 +
 +
Other execution fields (e.g. begin, id, lastUpdate) are ignored.
 +
 +
<source lang="xml">
 +
<execution>
 +
    <progress>0.5</progress>
 +
    <status>running</status>
 +
</execution>
 +
</source>
 +
 +
==== Adding a report ====
 +
 +
Create a new report for an existing task execution.
 +
 +
'''URL'''
 +
 +
*/api/tasks/{taskId}/executions/{executionId}/reports
 +
 +
'''Method'''
 +
 +
*POST
 +
 +
'''URL Params'''
 +
* Required:
 +
** taskId=[String]
 +
** executionId=[String]
 +
 +
'''Data Params'''
 +
 +
The body contains report metadata including:
 +
* the report name (String)
 +
* the report text (String)
 +
 +
<source lang="xml">
 +
<executionReport>
 +
  <name>out.log</name>
 +
  <text>report text...</text>
 +
</executionReport>
 +
</source>
 +
 +
==== Updating a report ====
 +
 +
Update an existing report.
 +
 +
'''URL'''
 +
 +
*/api/tasks/{taskId}/executions/{executionId}/reports/{reportName}
 +
 +
'''Method'''
 +
 +
*PUT
 +
 +
'''URL Params'''
 +
* Required:
 +
** taskId=[String]
 +
** executionId=[String]
 +
** reportName=[String]
 +
 +
'''Data Params'''
 +
 +
The body must contain the updated text of the report.
 +
* the report text (String)
 +
 +
<source lang="xml">
 +
<executionReport>
 +
  <text>udpated report text...</text>
 +
</executionReport>
 +
</source>
 +
 +
[[Category:CMEMS Dataset Importer]]

Latest revision as of 16:08, 6 March 2018

Overview

Usage

REST API

Discover the service

The service is registered in the Information System with the following coordinates:

Group: DataAnalysis

Name: cmems-importer-service

Browsing datasets

List Products

Request: 

GET /api/cmems/products

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products'

Sample output: 

<cmemsProducts>
  <product>
    <abstract>...</abstract>
    <allKeywords>
      numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
    </allKeywords>
    <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
    <creationDate>2016-10-14T00:00:00Z</creationDate>
    ...
  </product>
  <product>
    ...
  </product>
</cmemsProducts>

Get a Single Product

Request: 

GET /api/cmems/products/{productName}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024'

Sample output: 

<product>
  <abstract>...</abstract>
  <allKeywords>
    numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
  </allKeywords>
  <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
  <creationDate>2016-10-14T00:00:00Z</creationDate>
  ...
</product>

List Datasets available under the Subsetter protocol

Request: 

GET /api/cmems/products/{productName}/datasets/tds

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds'

Sample output: 

  <cmemsDatasets>
    <dataset>
      <name>global-analysis-forecast-phy-001-024</name>
      <protocol>TDS</protocol>
      <url>
        http://nrtcmems.mercator-ocean.fr/motu-web/Motu?action=describeproduct&service=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS&product=global-analysis-forecast-phy-001-024
      </url>
    </dataset>
    <dataset>
      ...
    <dataset>
  </cmemsDatasets>

Get Dataset details from the Motu server

Request: 

GET /api/cmems/products/{product}/datasets/tds/{dataset}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds/global-analysis-forecast-phy-001-024'

Sample output: 

  <productMetadataInfo code="0" id="global-analysis-forecast-phy-001-024" lastUpdate="Not Available" msg="OK" title="global-analysis-forecast-phy-001-024" url="">
    <dataGeospatialCoverage>
      <axis axisType="Time" code="0" description="Time (hours since 1950-01-01)" lower="499548" msg="OK" name="time" units="hours since 1950-01-01 00:00:00" upper="596820"/>
      <axis axisType="Lat" code="0" description="Latitude" lower="-80" msg="OK" name="latitude" units="degrees_north" upper="90"/>
      <axis axisType="Lon" code="0" description="Longitude" lower="-180" msg="OK" name="longitude" units="degrees_east" upper="179.9166717529296875"/>
      <axis axisType="Height" code="0" description="Depth" lower="0.4940249919891357421875" msg="OK" name="depth" units="m" upper="5727.9169921875"/>
    </dataGeospatialCoverage>
    <availableDepths>
      0.49402;1.54138;2.64567;3.81949;5.07822;...
    </availableDepths>
    <availableTimes>
      2006-12-27 12:00:00;2006-12-28 12:00:00;2006-12-29 12:00:00;...
    </availableTimes>
    ...
  </productMetadataInfo>

List Motu servers

To retrieve the list of all Motu servers publishing CMEMS data.

Request: 

GET /api/cmems/motu

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/motu'

Sample output: 

  <motuServers>
     <motuServer>
        <endpoint>
          http://cmems-bs-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     <motuServer>
        <endpoint>
          http://cmems-med-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     ...
  </motuServers>

List Geographical Regions

To retrieve the list of Geographical Regions covered by CMEMS data.

Request: 

GET /api/cmems/regions

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/regions'

Sample output: 

  <cmemsRegions>
    <cmemsRegion>
      <name>
        arctic-ocean
      </name>
    </cmemsRegion>
    <cmemsRegion>
      <name>
        baltic-sea
      </name>
    </cmemsRegion>
    ...
  </cmemsRegions>

List Variables

To retrieve the list of Variables present in CMEMS datasets.

Request: 

GET /api/cmems/variables

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/variables'

Sample output: 

  <cmemsVariables>
    <cmemsVariable>
      <name>
        age_of_sea_ice
      </name>
    </cmemsVariable>
    <cmemsVariable>
      <name>
        air_density
      </name>
    </cmemsVariable>
    ...
  </cmemsVariables>

Managing import tasks

Calculate the size of the imported dataset

To estimate the size, in bytes, of the imported dataset.

Request: 

GET /api/tasks/preview/sizes/dataset

URL parameters:

  • p: (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
  • d: (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
  • m: (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
  • v: (optional) the name of variables to import. Defaults to all variables in the dataset. Multiple parameters are allowed here.
  • tlo: (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
  • thi: (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
  • xlo: (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
  • xhi: (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
  • ylo: (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
  • yhi: (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
  • zlo: (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
  • zhi: (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024

Sample output: 

14451431889375

Calculate the size of a chunk of the imported dataset

Request: 

GET /api/tasks/preview/sizes/chunk

URL parameters:

  • All the parameters above, plus the following one:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&s=month

Sample output: 

  107206468022

Schedule an Import task

Request: 

POST /api/tasks

URL parameters: None

Data Params TODO

With curl: TODO

The operation returns the id of the scheduled task. Sample output: 

 ...

Browsing import tasks and reports

Get/search tasks

Return a list tasks, ordered by last execution time (or submit time, if not executed yet). Tasks can be filtered by user, token and scope, provided as query parameters.

Request: 

GET /api/tasks

URL parameters:

  • user: (optional) the name of the user who submitted the task.
  • token: (optional) the token used when submitting the task.
  • scope: (optional) the scope where the task was submitted.
  • limit: (optional) the maximum number of tasks returned.

Sample output: 

  <importTasks>
    <importTask>
      <submissionInfo>
        <scheduled>2018-01-10T19:28:59+02:00</scheduled>
        <scope>/gcube/devNext/NextNext</scope>
        <user>Alice</user>
      </submissionInfo>
      <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
      <lastExecution>
        <begin>2018-01-15T11:41:59+01:00</begin>
        <begin>2018-01-16T32:11:56+01:00</begin>
        <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
        <progress>1.0</progress>
        <reports>
          <report>
            <name>stderr.log</name>
            <size>25</size>
            <snippet>a few error messages here</snippet>
          </report>
          <report>
            <name>stdout.log</name>
            <size>3251</size>
            <snippet>
              Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum
              vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. 
              Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu
              odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
              quis sem elementum, dapib...
            </snippet>
          </report>
        </reports>
        <status>complete</status>
      </lastExecution>
      <importParameters>
        <dataset>datasetname</dataset>
        <motu>http://somehost/...</motu>
        <product>productname</product>
        <variables>
          <variable>salinity</variable>
          <variable>velocity</variable>
        </variables>
        ...
      </importParameters>
    </importTask>
    ...
  </importTask>

Retrieving a task

Return metadata about a single task, including submission details, import parameters, last execution info and reports snippets.

Request: 

GET /api/tasks/{taskId}

No parameters are expected in the URL.

Sample output: 

  <importTask>
    <submissionInfo>
      <scheduled>2018-01-10T19:28:59+02:00</scheduled>
      <scope>/gcube/devNext/NextNext</scope>
      <user>Alice</user>
    </submissionInfo>
    <id>69b529c1-6aac-48e0-9937-61fd28aa9e42</id>
    <lastExecution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>last</id>
      <progress>1.0</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...</snippet>
        </report>
      </reports>
      <status>complete</status>
    </lastExecution>
    <importParameters>
      <dataset>datasetname</dataset>
      <motu>http://somehost/...</motu>
      <product>productname</product>
      <variables>
        <variable>salinity</variable>
        <variable>velocity</variable>
      </variables>
      ...
    </importParameters>
  </importTask>

Retrieving task executions

Return a list of executions for a given task.

Request: 

GET /api/tasks/{taskId}/executions

No parameters are expected in the URL.

Sample output: 

  <executions>
    <execution>
      <begin>2018-01-15T11:41:59+01:00</begin>
      <begin>2018-01-16T32:11:56+01:00</begin>
      <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
      <progress>0.8725477544390946</progress>
      <reports>
        <report>
          <name>stderr.log</name>
          <size>25</size>
          <snippet>a few error messages here</snippet>
        </report>
        <report>
          <name>stdout.log</name>
          <size>3251</size>
          <snippet>
            Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
            Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
            viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
            lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
            efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
            tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
          </snippet>
        </report>
      </reports>
      <status>running</status>
    </execution>
    <execution>
      ...
    </execution>
  </executions>

Retrieving a task execution

Return a single execution for a given task.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}

No parameters are expected in the URL.

Sample output: 

  <execution>
    <begin>2018-01-15T11:41:59+01:00</begin>
    <begin>2018-01-16T32:11:56+01:00</begin>
    <id>33db1806-9830-4626-8b3b-c168c4be4fa7</id>
    <progress>0.8725477544390946</progress>
    <reports>
      <report>
        <name>stderr.log</name>
        <size>25</size>
        <snippet>a few error messages here</snippet>
      </report>
      <report>
        <name>stdout.log</name>
        <size>3251</size>
        <snippet>
          Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. 
          Pellentesque consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis 
          viverra. Phasellus vestibulum mauris non felis elementum molestie. Suspendisse vitae  
          lobortis ipsum. Vivamus egestas suscipit nibh. Donec ut venenatis purus. Maecenas 
          efficitur felis eu odio bibendum, id semper lectus feugiat. Suspendisse at fringilla 
          tellus, sit amet finibus orci. Aenean suscipit risus quis sem elementum, dapib...
        </snippet>
      </report>
    </reports>
    <status>running</status>
  </execution>

Retrieving execution reports

Return a list of execution reports (i.e. logs). For each report, a snippet of ~1000 characters is included.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports

No parameters are expected in the URL.

Sample output: 

  <executionReports>
    <executionReport>
      <name>stderr.log</name>
      <size>25</size>
      <snippet>a few error messages here</snippet>
    </executionReport>
    <executionReport>
      <name>stdout.log</name>
      <size>3251</size>
      <snippet>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
        quis sem elementum, dapib...
      </snippet>
    </executionReport>
  </executionReports>

Retrieving a single report

Retrieve the specified report.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}

No query parameters are expected in the URL.

Sample output: 

  <executionReports>
    <executionReport>
      <name>stderr.log</name>
      <size>25</size>
      <snippet>a few error messages here</snippet>
    </executionReport>
    <executionReport>
      <name>stdout.log</name>
      <size>3251</size>
      <snippet>
        Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
        consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
        vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
        suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
        lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
        quis sem elementum, dapib...
      </snippet>
    </executionReport>
  </executionReports>

Retrieving report text only

Retrieve only the text contained in the specified report.

Request: 

GET /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}/text

No query parameters are expected in the URL.

Sample output: 

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vitae vulputate sapien. Pellentesque
consequat ex quis dictum vulputate. Nunc eleifend arcu et elit sagittis viverra. Phasellus
vestibulum mauris non felis elementum molestie. Suspendisse vitae lobortis ipsum. Vivamus egestas
suscipit nibh. Donec ut venenatis purus. Maecenas efficitur felis eu odio bibendum, id semper
lectus feugiat. Suspendisse at fringilla tellus, sit amet finibus orci. Aenean suscipit risus 
quis sem elementum, dapib...

Updating executions and reports

This interface is meant to be used by the cmems-importer-se-plugin, while running the import task.

Creating a new execution

Create a new execution for an existing import task.

URL

  • /api/tasks/{taskId}/executions

Method

  • POST

URL Params

  • Required:
    • taskId=[String]

Data Params

The body contains execution metadata including:

  • the execution id (String)
  • the current status (String)
  • execution progress (Double within [0,1])
  • the execution start time
  • the execution end time
<execution>
    <id>1517927891274</id>
    <begin>2018-02-06T15:38:13.579+01:00</begin>
    <progress>0.1</progress>
    <status>running</status>
</execution>

Updating an execution

Update an existing execution.

URL

  • /api/tasks/{taskId}/executions/{executionId}

Method

  • PUT

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]

Data Params

The body contains execution metadata including:

  • the current status (String)
  • execution progress (Double within [0,1])
  • the execution end time

Other execution fields (e.g. begin, id, lastUpdate) are ignored. 

<execution>
    <progress>0.5</progress>
    <status>running</status>
</execution>

Adding a report

Create a new report for an existing task execution.

URL

  • /api/tasks/{taskId}/executions/{executionId}/reports

Method

  • POST

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]

Data Params

The body contains report metadata including:

  • the report name (String)
  • the report text (String)
<executionReport>
  <name>out.log</name>
  <text>report text...</text>
</executionReport>

Updating a report

Update an existing report.

URL

  • /api/tasks/{taskId}/executions/{executionId}/reports/{reportName}

Method

  • PUT

URL Params

  • Required:
    • taskId=[String]
    • executionId=[String]
    • reportName=[String]

Data Params

The body must contain the updated text of the report.

  • the report text (String)
<executionReport>
  <text>udpated report text...</text>
</executionReport>
Retrieved from "https://wiki.gcube-system.org/index.php?title=CMEMS_Importer_Service&oldid=31014"