CMEMS Importer Client

Overview

The CMEMS Importer client library allows the interaction with SmartExecutor framework to execute, monitor, schedule, stop an import task, as well as list the currently scheduled tasks, with their import parameters. This library is part of the CMEMS Dataset Importer.

Usage

Configure your project

Maven coordinates:

  <dependency>
    <groupId>org.gcube.dataanalysis</groupId>
    <artifactId>cmems-importer-client</artifactId>
    <version>[1.0.0, 2.0.0)</version>
  </dependency>

The current version of the library is 1.0.0

Create a client

A client is simply created with:

  CmemsImporterClient client = new CmemsImporterClient();

Scheduling the import of a new dataset

To schedule the import of a dataset, details about what to import have to be given. This is done using the ImportOption class.

  ImportOptions options = new ImportOptions();

Then, import options can be given with:

  // The endpoint of the motu server as provided by the CMEMS Client. Mandatory field.
  options.setMotu("http://endpoint/of/the/motu/server");
 
  // The name of the product. Mandatory field.  options.setProduct("GLOBAL_ANALYSIS_FORECAST_PHY_001_024");
 
  // The name of the dataset. Mandatory field.
  options.setDataset("global-analysis-forecast-phy-001-024-hourly-t-u-v-ssh");
 
  // Set the time span of the imported dataset.
  // Optional fields; default to dataset boundaries.
  Calendar from = ...;
  Calendar to = ...;
  options.settLo(from);
  options.settHi(to);
 
  // Set the longitude range of the imported dataset.
  // Optional fields; default to dataset boundaries.
  Double minLong = ...;
  Double maxLong = ...;
  options.setxLo(minLong);
  options.setxHi(maxLong);
 
  // Set the latitude range of the imported dataset.
  // Optional fields; default to dataset boundaries.
  Double minLat = ...;
  Double maxLat = ...;
  options.setyLo(minLat);
  options.setyHi(maxLat);
 
  // Set the depth range of the imported dataset.
  // Optional fields; default to dataset boundaries.
  Double minDepth = ...;
  Double maxDepth = ...;
  options.setzLo(minDepth);
  options.setzHi(maxDepth);
 
  // The variables to import.
  // Optional field; defaults to all variables in the dataset.
  options.addVariable("vgos");
  options.addVariable("sla");
 
  // The time span of the imported dataset chunk. Can be 'month' or 'year'. Mandatory field.
  options.setChunkSpan("month");
 
  // How many days in the past data should not be imported, wrt the import execution time.
  // Optional field; defaults to 0.
  options.setBackTime(7);
 
  // How often the import task should run.
  // The importer will not execute more than import task per day, per scheduled task.
  // Optional; defaults to 'weekly' at a random day and time, to spread import tasks evenly along time.
  options.setImportSchedule(String cronExpression);

Once the import options are set, values can be checked for consistency against the remote dataset. In case validation fails, the method throws exceptions.

  client.checkOptions();

Finally, the import task can be scheduled with:

  client.scheduleTask(options);

For user information, he size of the whole imported dataset, the size of the dataset chunk and the size of periodic update can be computed:

  // this is the size of the whole imported dataset
  Long datasetSize = client.getDatasetSize(options);
 
  // this is the size of a dataset chunk (e.g. the monthly or yearly files)  Long chunkSize = client.getChunkSize(options);
 
  // this is the size of the dataset portion downloaded periodically (i.e. the diff)
  Long updateSize = client.getDiffSize(options);

Listing currently-scheduled tasks

A list of currently-schedule tasks can be obtained with:

  Collection<String> ids = client.listTasks();

A set of execution IDs is returned. Schedule options for each task can be obtained with:

  String taskId = ...;
  ImportOptions options = client.getTaskOptions(taskId);

Note: this functionality is not yet ready as it depends on the corresponding API on the SmartExecutor side.

Stopping a scheduled task

An import task currently scheduled for import, can be stopped with:

  String taskId = ...;
  unscheduleTask(taskId);

Getting the status of a scheduled task

The importer client surfaces the methods of the SmartExecutor client to retrieve the status of a running task:

  import org.gcube.vremanagement.executor.plugin.PluginStateEvolution;
 
  String taskId = ...;
  PluginStateEvolution status = client.getTaskStatus(taskId);