Difference between revisions of "SmartExecutor"

From Gcube Wiki
Jump to: navigation, search
(Plugin Development)
(Monitor Task Execution)
Line 70: Line 70:
 
The possible state for a task are:
 
The possible state for a task are:
  
* CREATED : The Task is created but not still running.
+
* '''''CREATED''''' : The Task is created but not still running.
* RUNNING : The Task is running.
+
* '''''RUNNING''''' : The Task is running.
* STOPPED : The Task has been stopped.
+
* '''''STOPPED''''' : The Task has been stopped.
* DONE : The Task terminated successfully.
+
* '''''DONE''''' : The Task terminated successfully.
* FAILED : The Task failed the execution.
+
* '''''FAILED''''' : The Task failed the execution.
* DISCARDED : The Task has been discarded by the scheduler. This happen only for repetitive or recurrent tasks and only when the launch parameter require that the previous task must be completed.
+
* '''''DISCARDED''''' : The Task has been discarded by the scheduler. This happen only for repetitive or recurrent tasks and only when the launch parameter require that the previous task must be completed.
  
 
The are two API to monitor a task execution.
 
The are two API to monitor a task execution.

Revision as of 10:33, 21 October 2015

The SmartExecutor service allows to execute "gCube Tasks" and monitor their execution status. Each instance of the SmartExecutor service can run the "gCube Tasks" related to the plugins available on such an instance.

Each instance of the SmartExecutor service publishes descriptive information about the co-deployed plugins.

Clients may interact with the SmartExecutor service through a library (SmartExecutor Client) of high-level facilities to simplify the discovery of available plugins in those instances. Each client can request to execute a "gCube Tasks" or getting informations about the state of their execution.

The Service

SmartExecutor service allows tasks execution through the use of co-deployed plugins. The service allows to pass inputs parameter to the plugin requested to run.

SmartExecutor service can run the tasks in two mode:

  • Run and Die
  • Scheduled

In Run and Die mode the requested plugin has been run just one time using the input parameters contained in the invocation request. When the execution terminate because is finished, is stopped or stop due to exception a new execution can be run only if is explicitly requested.

In Scheduled mode the execution is invoked every time it matches the scheduling parameters. The way to schedule the plugin execution is indicated by scheduling parameter (see Launch Inputs).

Plugins

At startup time the SmartExecutor discover the available plugins (using java.util.ServiceLoader). For each discovered plugin the SmartExecutor gather some mandatory information (e.g. version, name) and ask to the plugin to discover the capabilities of the machine. The discovered capabilities can be used by clients to select one instance instead of another (e.g. available python version, available R version, a certain hardware capability). The machine capabilities are plugin dependend so that is responsibility of the plugin to discover them.

The SmartExecutor publish the mandatory information and the discovered capabilities on IS using a ServiceEndpoint with Category: VREManagement and Name: SmartExecutor.

The SmartExecutor Client search for this service endpoints to retrieve available SmartExecutor instances running a certain plugin.

Client

The SmartExecutor service provides a client library to simplify the following procedures:

  • discover service instances that can execute the target task. This requires interaction with the Information System.
  • launch the execution of the task with one the discovered instances.
  • monitor the execution of the running task.

Instance Discovery

SmartExecutor Client provide some facilities to get the SmartExecutor service instances.

It allows to retrieve one of the the running SmartExecutor services filtering them in the following way:

  • By plugin name only
  • By plugin name AND one or more of the following filters:
    • KEY-VALUE list. Each KEY-VALUE must be found on the Service Endpoints.
    • XQuery conditions to filter the Service Endpoints. The conditions are added to the standard query performed.
    • List of endpoints to be selected within the available one.


Using SmartExecutor Client a proxy for remote invocation can be obtained. Using this proxy is very easy to call a remote method as the method was local.

Launching Tasks

One of the API exposed by the SmartExecutor service is used to launch a plugin execution instance. Each time a task launch is executed the service (after some checks) create a separated thread, instantiate the related plugin and invoke the execution passing inputs parameters. The service return immediately after the task is launched (task is executed asynchronously) The string representation of the ID (an UUID) is retuned to the clients. That ID can be used to monitor the task execution.

Launch Inputs

The invocation parameters are:

  • name of the plugin to be instantiated and run (MANDATORY)
  • inputs to pass to plugin instance execution (MANDATORY)
  • capabilities the plugin has to satisfy to correctly run an execution (e.g. the machine has a certain version of python interpreter) (OPTIONAL)
  • scheduling strategy to launch the execution (e.g. every day at 22:00) (OPTIONAL)
  • persist if the scheduling has to be persisted. If the SmartExecutor instance terminate another SmartExecutor must take in charge that scheduling. Due to a scheduling can survive over the time is strongly recommend to declare the needed capabilities.

Please note that the capabilities matches are checked before launching the plugin execution. This is a service check and is conceptually different from filtering in instance discovery using the client.

Monitor Task Execution

The ID returned invoking the launch can be used to monitor an execution.

The possible state for a task are:

  • CREATED : The Task is created but not still running.
  • RUNNING : The Task is running.
  • STOPPED : The Task has been stopped.
  • DONE : The Task terminated successfully.
  • FAILED : The Task failed the execution.
  • DISCARDED : The Task has been discarded by the scheduler. This happen only for repetitive or recurrent tasks and only when the launch parameter require that the previous task must be completed.

The are two API to monitor a task execution.

  • The first one to retrieve the current state of the last execution.
  • The second one to retrieve the current state of the nth execution (execution number).

The latter is used to monitor scheduled tasks. The execution number is used to identify the nth run launched respecting the scheduling strategy.

To monitor the run and die tasks the first API should be used. You could use the second API providing the value 1 for (execution number).

Requesting the state for en execution that has not been performed raise an exception.

Plugin Development

The simplest way to start to develop a plugin is to checkout the HelloWorld plugin

https://svn.d4science.research-infrastructures.eu/gcube/trunk/vre-management/smart-executor-HelloWorldPlugin

Each plugin must declare:

  • Name
  • Version
  • Description

Moreover the plugin is able to discover the machine capabilities it wants to publish, to allows clients to filters instances, by providing them as a key-value map

The mandatory information and the discovered capabilities are published in the ServiceEndpoint by the SmartExecutor service.

SmartExecutor plugins may have arbitrary size and dependencies.

Each plugin must implements at least two classes:

  • The first one has to extend the org.gcube.vremanagement.executor.plugin.PluginDeclaration interface. This class is responsible of providing the mandatory information and discover the capabilities. this class is used by the SmartExecutor service just one time at application startup, to populate the ServiceEndpoint.
  • The second class had to implement org.gcube.vremanagement.executor.plugin.Plugin<? extends PluginDeclaration> class. This class must implements two method. The first one responsible to launch the plugin execution with the provided input parameters. The second method is used to perform termination actions when an execution is stopped due to a client request.