OscarImporterSEPlugin

The Oscar-Importer-SE-Plugin is a Smart Executor plugin which periodically (currently every week) builds a merged version of the OSCAR dataset and uploads it to the infrastructures Thredds server.

Following user requirements, the importer is not using the original dataset, but a version where coordinates have been rearranged into -180 to 180 instead of 20 to 420 like the other OSCAR datasets

Deployment

The plugin has to be deployed on a Smart Executor node with at least 25GB of free disk space in /tmp

On the smart executor node, copy the jar-with-dependencies in /home/gcube/tomcat/webapps/smart-executor/WEB-INF/lib

Managing the service

The Smart executor needs to be restarted in order to discover and publish the plugin in the Information System.

stop the container

 ./stopContainer.sh  

start the container

 ./startContainer.sh

Check

• check there are no exception in:
  • SmartGears/ghn.log
  • tomcat/logs/catalina.out
  • tomcat/logs/localhost.log
• check that the profile are published in the IS; look at the infrastructure monitor (Service Endpoint -> VREManagement)

Starting the job

One shot

 TODO

Scheduled

 TODO

Under the hood

• To save disk space, the importer downloads small groups (currently 3) of yearly oscar files, merges them and proceeds iteratively.

• The merger uses /tmp/oscar-merger as working directory. All needed files (downloaded, merged, descriptors, temporary) are placed here. No other directory is used on the machine.

• Upon successful upload to Thredds, the work directory is cleaned with the exception of the last stable merged file (e.g. oscar-1992-2016.nc); this will speed-up next merge since only diff files (e.g. current year) will be downloaded and merged to it.