Revision as of 11:18, 7 July 2017

The Oscar-Importer-SE-Plugin is a Smart Executor plugin which periodically (currently every week) builds a merged version of the OSCAR ^[1] dataset and uploads it to the infrastructures Thredds server.

Following user requirements, the importer is not using the original dataset, but a version ^[2] where coordinates have been rearranged into -180 to 180 instead of 20 to 420 like the other OSCAR datasets

Deployment

The plugin has to be deployed on a Smart Executor node with at least 25GB of free disk space in /tmp. The disk space is required for files to be merged and for the merged file before uploading.

On the smart executor node, copy the jar-with-dependencies in /home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib

Managing the service

The Smart executor needs to be restarted in order to discover and publish the plugin in the Information System.

Stop the container:

 ./stopContainer.sh  

Start the container:

 ./startContainer.sh

Check

• check there are no exception in:

  SmartGears/ghn.log
  tomcat/logs/catalina.out
  tomcat/logs/localhost.log

• check that the profile are published in the IS; look at the infrastructure monitor (Service Endpoint -> VREManagement)

Starting the job

One shot

 TODO

Scheduled

 TODO

Under the hood

• To save disk space, the importer downloads small groups (currently 3) of yearly oscar files, merges them and proceeds iteratively.

• The merger uses /tmp/oscar-merger as working directory. All needed files (downloaded, merged, descriptors, temporary) are placed here. No other directory is used on the machine.

• Upon successful upload to Thredds, the work directory is cleaned with the exception of the last stable merged file (e.g. oscar-1992-2016.nc); this will speed-up next merge since only diff files (e.g. current year) will be downloaded and merged to it.

References

1. ↑ https://podaac.jpl.nasa.gov/dataset/OSCAR_L4_OC_third-deg
2. ↑ ftp://podaac-ftp.jpl.nasa.gov/allData/oscar/preview/L4/resource/LAS/oscar_third_deg_180/