Difference between revisions of "CMEMS Importer Service"

From Gcube Wiki
Jump to: navigation, search
(→‎REST API)
(→‎REST API)
Line 45: Line 45:
 
</source>
 
</source>
  
−
===Get a Single Product===
+
====Get a Single Product====
  
 
Request:
 
Request:
Line 70: Line 70:
 
</source>
 
</source>
  
−
===List Datasets available under the Subsetter protocol===
+
====List Datasets available under the Subsetter protocol====
  
 
Request:
 
Request:
Line 99: Line 99:
 
</source>
 
</source>
  
−
===Get Dataset details from the Motu server===
+
====Get Dataset details from the Motu server====
  
 
Request:
 
Request:
Line 130: Line 130:
 
</source>
 
</source>
  
−
===List Motu servers===
+
====List Motu servers====
  
 
To retrieve the list of all Motu servers publishing CMEMS data.
 
To retrieve the list of all Motu servers publishing CMEMS data.
Line 161: Line 161:
 
</source>
 
</source>
  
−
===List Geographical Regions===
+
====List Geographical Regions====
  
 
To retrieve the list of Geographical Regions covered by CMEMS data.
 
To retrieve the list of Geographical Regions covered by CMEMS data.
Line 192: Line 192:
 
</source>
 
</source>
  
−
===List Variables===
+
====List Variables====
  
 
To retrieve the list of Variables present in CMEMS datasets.
 
To retrieve the list of Variables present in CMEMS datasets.
Line 223: Line 223:
 
</source>
 
</source>
  
−
===Calculate the size of the imported dataset===
+
===Managing import tasks===
 +
 
 +
====Calculate the size of the imported dataset====
  
 
To estimate the size, in bytes, of the imported dataset.
 
To estimate the size, in bytes, of the imported dataset.
Line 261: Line 263:
 
</source>
 
</source>
  
−
===Calculate the size of a chunk of the imported dataset===
+
====Calculate the size of a chunk of the imported dataset====
  
 
Request:
 
Request:
Line 283: Line 285:
 
</source>
 
</source>
  
−
===OP===
+
====Schedule an Import task====
  
 
Request:
 
Request:
 
<source lang="java">
 
<source lang="java">
−
GET /api/
+
GET /api/tasks/schedule
 
</source>
 
</source>
  
 
URL parameters:
 
URL parameters:
−
* '''v''': desc here
+
* All the parameters above, plus the following ones:
 +
* '''s''': (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.
 +
* '''b''': (optional) how many days in the past data should not be imported, wrt the import execution time. If not set, defaults to 0.
 +
* '''f''': (mandatory) a cron-compliant expression to set how often the import task should run. The importer will not execute more than import task per day, per scheduled task.
  
 
With curl:
 
With curl:
 
<source lang="php">
 
<source lang="php">
 +
$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/schedule?p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&s=month
 
</source>
 
</source>
  

Revision as of 13:14, 22 January 2018

Overview

Usage

REST API

Discover the service

The service is registered in the Information System with the following coordinates:

Group: DataAnalysis

Name: cmems-importer-service

Browsing datasets

List Products

Request: 

GET /api/cmems/products

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products'

Sample output: 

<cmemsProducts>
  <product>
    <abstract>...</abstract>
    <allKeywords>
      numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
    </allKeywords>
    <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
    <creationDate>2016-10-14T00:00:00Z</creationDate>
    ...
  </product>
  <product>
    ...
  </product>
</cmemsProducts>

Get a Single Product

Request: 

GET /api/cmems/products/{productName}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024'

Sample output: 

<product>
  <abstract>...</abstract>
  <allKeywords>
    numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
  </allKeywords>
  <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
  <creationDate>2016-10-14T00:00:00Z</creationDate>
  ...
</product>

List Datasets available under the Subsetter protocol

Request: 

GET /api/cmems/products/{productName}/datasets/tds

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds'

Sample output: 

  <cmemsDatasets>
    <dataset>
      <name>global-analysis-forecast-phy-001-024</name>
      <protocol>TDS</protocol>
      <url>
        http://nrtcmems.mercator-ocean.fr/motu-web/Motu?action=describeproduct&service=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS&product=global-analysis-forecast-phy-001-024
      </url>
    </dataset>
    <dataset>
      ...
    <dataset>
  </cmemsDatasets>

Get Dataset details from the Motu server

Request: 

GET /api/cmems/products/{product}/datasets/tds/{dataset}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds/global-analysis-forecast-phy-001-024'

Sample output: 

  <productMetadataInfo code="0" id="global-analysis-forecast-phy-001-024" lastUpdate="Not Available" msg="OK" title="global-analysis-forecast-phy-001-024" url="">
    <dataGeospatialCoverage>
      <axis axisType="Time" code="0" description="Time (hours since 1950-01-01)" lower="499548" msg="OK" name="time" units="hours since 1950-01-01 00:00:00" upper="596820"/>
      <axis axisType="Lat" code="0" description="Latitude" lower="-80" msg="OK" name="latitude" units="degrees_north" upper="90"/>
      <axis axisType="Lon" code="0" description="Longitude" lower="-180" msg="OK" name="longitude" units="degrees_east" upper="179.9166717529296875"/>
      <axis axisType="Height" code="0" description="Depth" lower="0.4940249919891357421875" msg="OK" name="depth" units="m" upper="5727.9169921875"/>
    </dataGeospatialCoverage>
    <availableDepths>
      0.49402;1.54138;2.64567;3.81949;5.07822;...
    </availableDepths>
    <availableTimes>
      2006-12-27 12:00:00;2006-12-28 12:00:00;2006-12-29 12:00:00;...
    </availableTimes>
    ...
  </productMetadataInfo>

List Motu servers

To retrieve the list of all Motu servers publishing CMEMS data.

Request: 

GET /api/cmems/motu

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/motu'

Sample output: 

  <motuServers>
     <motuServer>
        <endpoint>
          http://cmems-bs-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     <motuServer>
        <endpoint>
          http://cmems-med-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     ...
  </motuServers>

List Geographical Regions

To retrieve the list of Geographical Regions covered by CMEMS data.

Request: 

GET /api/cmems/regions

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/regions'

Sample output: 

  <cmemsRegions>
    <cmemsRegion>
      <name>
        arctic-ocean
      </name>
    </cmemsRegion>
    <cmemsRegion>
      <name>
        baltic-sea
      </name>
    </cmemsRegion>
    ...
  </cmemsRegions>

List Variables

To retrieve the list of Variables present in CMEMS datasets.

Request: 

GET /api/cmems/variables

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/variables'

Sample output: 

  <cmemsVariables>
    <cmemsVariable>
      <name>
        age_of_sea_ice
      </name>
    </cmemsVariable>
    <cmemsVariable>
      <name>
        air_density
      </name>
    </cmemsVariable>
    ...
  </cmemsVariables>

Managing import tasks

Calculate the size of the imported dataset

To estimate the size, in bytes, of the imported dataset.

Request: 

GET /api/tasks/preview/sizes/dataset

URL parameters:

  • p: (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
  • d: (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
  • m: (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
  • v: (optional) the name of variables to import. Defaults to all variables in the dataset. Multiple parameters are allowed here.
  • tlo: (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
  • thi: (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
  • xlo: (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
  • xhi: (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
  • ylo: (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
  • yhi: (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
  • zlo: (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
  • zhi: (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024

Sample output: 

14451431889375

Calculate the size of a chunk of the imported dataset

Request: 

GET /api/tasks/preview/sizes/chunk

URL parameters:

  • All the parameters above, plus the following one:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/preview/sizes/dataset?xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&s=month

Sample output: 

  107206468022

Schedule an Import task

Request: 

GET /api/tasks/schedule

URL parameters:

  • All the parameters above, plus the following ones:
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.
  • b: (optional) how many days in the past data should not be imported, wrt the import execution time. If not set, defaults to 0.
  • f: (mandatory) a cron-compliant expression to set how often the import task should run. The importer will not execute more than import task per day, per scheduled task.

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/tasks/schedule?p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024&xlo=-5&xhi=5&ylo=-5&yhi=5&tlo=2010&thi=2012&s=month

Sample output: 

 
Retrieved from "https://wiki.gcube-system.org/index.php?title=CMEMS_Importer_Service&oldid=30819"