Difference between revisions of "CMEMS Importer Service"

From Gcube Wiki
Jump to: navigation, search
(→‎OP)
(→‎OP)
Line 221: Line 221:
 
</source>
 
</source>
  
−
===OP===
+
===Calculate the size of the imported dataset===
 +
 
 +
To estimate the size, in bytes, of the imported dataset.
  
 
Request:
 
Request:
 
<source lang="java">
 
<source lang="java">
−
GET /api/
+
GET /api/tasks/preview/sizes/dataset
 
</source>
 
</source>
  
 
URL parameters:
 
URL parameters:
−
* '''v''': desc here
+
* '''p''': (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
 +
* '''d''': (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
 +
* '''m''': (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
 +
* '''v''': (optional) the name of variables to import. Defaults to all variables in the dataset.
 +
* '''tlo''': (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
 +
* '''thi''': (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
 +
* '''xlo''': (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
 +
* '''xhi''': (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
 +
* '''ylo''': (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
 +
* '''yhi''': (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
 +
* '''zlo''': (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
 +
* '''zhi''': (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.
 +
 
 +
* '''s''': (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.
 +
* '''b''': (optional) how many days in the past data should not be imported, wrt the import execution time. If not set, defaults to 0.
 +
* '''f''': (mandatory) a cron-compliant expression to set how often the import task should run. The importer will not execute more than import task per day, per scheduled task.
 +
 
  
 
With curl:
 
With curl:
 
<source lang="php">
 
<source lang="php">
 +
  http://localhost:8080/cmems-importer-service/rest/tasks/preview/sizes/dataset?p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024
 
</source>
 
</source>
  
 
Sample output:
 
Sample output:
−
<source lang="xml">
+
<source lang="java">
 +
14451431889375
 
</source>
 
</source>
  

Revision as of 13:01, 22 January 2018

Overview

Usage

REST API

Discover the service

The service is registered in the Information System with the following coordinates:

Group: DataAnalysis

Name: cmems-importer-service

List Products

Request: 

GET /api/cmems/products

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/rest/cmems/products'

Sample output: 

<cmemsProducts>
  <product>
    <abstract>...</abstract>
    <allKeywords>
      numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
    </allKeywords>
    <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
    <creationDate>2016-10-14T00:00:00Z</creationDate>
    ...
  </product>
  <product>
    ...
  </product>
</cmemsProducts>

Get a Single Product

Request: 

GET /api/cmems/products/{productName}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/rest/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024'

Sample output: 

<product>
  <abstract>...</abstract>
  <allKeywords>
    numerical-model, salinity, sea-ice, sea-level, currents, temperature, forecast, near-real-time, global-ocean
  </allKeywords>
  <coordRefSys>WGS 84 (EPSG 4326)</coordRefSys>
  <creationDate>2016-10-14T00:00:00Z</creationDate>
  ...
</product>

List Datasets available under the Subsetter protocol

Request: 

GET /api/cmems/products/{productName}/datasets/tds

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/rest/cmems/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds'

Sample output: 

  <cmemsDatasets>
    <dataset>
      <name>global-analysis-forecast-phy-001-024</name>
      <protocol>TDS</protocol>
      <url>
        http://nrtcmems.mercator-ocean.fr/motu-web/Motu?action=describeproduct&service=GLOBAL_ANALYSIS_FORECAST_PHY_001_024-TDS&product=global-analysis-forecast-phy-001-024
      </url>
    </dataset>
    <dataset>
      ...
    <dataset>
  </cmemsDatasets>

Get Dataset details from the Motu server

Request: 

GET /api/cmems/products/{product}/datasets/tds/{dataset}

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/products/GLOBAL_ANALYSIS_FORECAST_PHY_001_024/datasets/tds/global-analysis-forecast-phy-001-024'

Sample output: 

  <productMetadataInfo code="0" id="global-analysis-forecast-phy-001-024" lastUpdate="Not Available" msg="OK" title="global-analysis-forecast-phy-001-024" url="">
    <dataGeospatialCoverage>
      <axis axisType="Time" code="0" description="Time (hours since 1950-01-01)" lower="499548" msg="OK" name="time" units="hours since 1950-01-01 00:00:00" upper="596820"/>
      <axis axisType="Lat" code="0" description="Latitude" lower="-80" msg="OK" name="latitude" units="degrees_north" upper="90"/>
      <axis axisType="Lon" code="0" description="Longitude" lower="-180" msg="OK" name="longitude" units="degrees_east" upper="179.9166717529296875"/>
      <axis axisType="Height" code="0" description="Depth" lower="0.4940249919891357421875" msg="OK" name="depth" units="m" upper="5727.9169921875"/>
    </dataGeospatialCoverage>
    <availableDepths>
      0.49402;1.54138;2.64567;3.81949;5.07822;...
    </availableDepths>
    <availableTimes>
      2006-12-27 12:00:00;2006-12-28 12:00:00;2006-12-29 12:00:00;...
    </availableTimes>
    ...
  </productMetadataInfo>

List Motu servers

To retrieve the list of all Motu servers publishing CMEMS data.

Request: 

GET /api/cmems/motu

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/motu'

Sample output: 

  <motuServers>
     <motuServer>
        <endpoint>
          http://cmems-bs-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     <motuServer>
        <endpoint>
          http://cmems-med-mfc.eu/motu-web/Motu
        </endpoint>
     </motuServer>
     ...
  </motuServers>

List Geographical Regions

To retrieve the list of Geographical Regions covered by CMEMS data.

Request: 

GET /api/cmems/regions

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/regions'

Sample output: 

  <cmemsRegions>
    <cmemsRegion>
      <name>
        arctic-ocean
      </name>
    </cmemsRegion>
    <cmemsRegion>
      <name>
        baltic-sea
      </name>
    </cmemsRegion>
    ...
  </cmemsRegions>

List Variables

To retrieve the list of Variables present in CMEMS datasets.

Request: 

GET /api/cmems/variables

With curl: 

$ curl --header "gcube-token: **********" --request GET 'http://<hostname>/cmems-importer-service/api/cmems/variables'

Sample output: 

  <cmemsVariables>
    <cmemsVariable>
      <name>
        age_of_sea_ice
      </name>
    </cmemsVariable>
    <cmemsVariable>
      <name>
        air_density
      </name>
    </cmemsVariable>
    ...
  </cmemsVariables>

Calculate the size of the imported dataset

To estimate the size, in bytes, of the imported dataset.

Request: 

GET /api/tasks/preview/sizes/dataset

URL parameters:

  • p: (mandatory) the name of the product (e.g. GLOBAL_ANALYSIS_FORECAST_PHY_001_024).
  • d: (mandatory) the name of the dataset (e.g. global-analysis-forecast-phy-001-024).
  • m: (optional) the endpoint of the Motu server publishing the given dataset. If not given, the service will search for it with an extra call to CMEMS.
  • v: (optional) the name of variables to import. Defaults to all variables in the dataset.
  • tlo: (optional) start date of a temporal extraction - e.g. 2017-01-24. If not set, the default value is the first date/time available for the dataset.
  • thi: (optional) end date of a temporal extraction - e.g. 2017-07-24. If not set, the default value is the last date/time available for the dataset.
  • xlo: (optional) low longitude of the geographic extraction. The default value is the lowest longitude in the dataset.
  • xhi: (optional) high longitude of the geographic extraction. The default value is the highest longitude in the dataset.
  • ylo: (optional) low latitude of the geographic extraction. The default value is the lowest latitude in the dataset.
  • yhi: (optional) high latitude of the geographic extraction. The default value is the highest latitude in the dataset.
  • zlo: (optional) low depth of the geographic extraction. The default value is the lowest depth in the dataset.
  • zhi: (optional) high depth of the geographic extraction. The default value is the highest depth in the dataset.
  • s: (optional) the time span of the imported dataset chunk. Can be 'month' or 'year'. If not set, the default value is 'month'.
  • b: (optional) how many days in the past data should not be imported, wrt the import execution time. If not set, defaults to 0.
  • f: (mandatory) a cron-compliant expression to set how often the import task should run. The importer will not execute more than import task per day, per scheduled task.


With curl: 

  http://localhost:8080/cmems-importer-service/rest/tasks/preview/sizes/dataset?p=GLOBAL_ANALYSIS_FORECAST_PHY_001_024&d=global-analysis-forecast-phy-001-024

Sample output: 

14451431889375

OP

Request: 

GET /api/

URL parameters:

  • v: desc here

With curl: 

 

Sample output: 

 
Retrieved from "https://wiki.gcube-system.org/index.php?title=CMEMS_Importer_Service&oldid=30805"