ExecutionPlan Elements

From Gcube Wiki
Revision as of 10:25, 10 February 2010 by Giorgos.papanikos (Talk | contribs) (→‎Processing)

Jump to: navigation, search

Overview

All plan elements are automatically assigned with an unique identifier through which they are retrievable and identified. Additionally, all elements can be assigned with a name that they use when emitting progress events. This way the execution monitoring can be made more user friendly.

The elements listed below can be roughly separated in two large categories based on the position they may have in the execution tree.

All the Nodes elements have placeholders where other elements, either Nodes themselves or Leaves can be plugged in. The Leaves on the other hand can be plugged in other Nodes but the execution plan path ends to them.

Plan Elements

Sequence

The Sequence construct offers a simple ordering utility for a set of execution steps. This element is a simple container maintaining the order of execution of its sub elements. When executed, the ordered list of sub elements is executed one after the other and the Sequence element is completed when its last sub-element is completed.

XML definition

<planElement id="element id" name="element name" type="Sequence">
  <list>
    <planElement .../>
    <planElement .../>
    <planElement .../>
    ...
  </list>
</planElement>

Flow

The Flow construct offers a simple parallelization utility for a set of execution steps. This element is a simple container maintaining a set of sub elements that can be executed in parallel. When executed, the set of sub elements are executed all at once in parallel and the Flow only completed when all the sub elements that have started are terminated. If one of the elements is terminated with an error, after all the elements have terminated, the exception indicating the error is re-thrown. If more than one elements finished with an error, only the first error found iterating through the set of sub elements is thrown.

XML definition

<planElement id="element id" name="element name" type="Flow">
  <list>
    <planElement .../>
    <planElement .../>
    <planElement .../>
    ...
  </list>
</planElement>

Bag

The Bag construct acts as a container of other sub elements. It orchestrates their execution based on their execution preconditions. These elements as well as their conditions are evaluated every time one of the previously running elements have completed their execution. While this element is executing every time an element previously running has terminated the set of not yet run elements is scanned. If no element can be run while no more elements are still running, the process is terminated if such a behavior is requested. Even if more elements are still running, if termination condition is set and is evaluated to true, the execution of the element is terminated. This condition follows the syntax defined by Execution Conditions. The execution preconditions of each element also follow the syntax defined by Execution Conditions.

XML definition

<planElement id="element id" name="element name" type="Bag">
  <list>
    <bagElement>
      <planElement ... />
      <conditionTree .../>
    </bagElement>
    <bagElement>
      <planElement ... />
      <conditionTree .../>
    </bagElement>
    ...
  </list>
  <termination onNoProgress="true">
    <conditionTree .../>
  </termination>
</planElement>

Know issues

It is possible that if it has not been requested for the Bag to terminate when no progress can be made, and the termination condition, if set, does not cover this case, that the Bag element will never terminate its execution.

Furthermore, currently the execution conditions of the bag's elements are only evaluated once at the beginning of the execution and then every time a previously running element terminates. This means that even if some execution condition is not dependent of some other element being terminated, it will not be discovered before that.

File Transfer

The File Tranfer construct can be used to mediate at the execution level the staging of files to and from the StorageSystem. The transfer can have a direction of either Store or Retrieve. In case the direction is Store, the File Tranfer utility expects as input the file path that is to be stored in the StorageSystem and is located in the same hosting machine as the one executing the element. After the operation is completed, the output of the File Transfer will be the identifier assigned to the stored file by the StorageSystem. In case the direction is Retrieve, the input is expected to be the identifier of the document that is stored in the StorageSystem and after the execution, the output of Filte Transfer will have the local filename where the document is stored.

When the File Transfer direction is Retrieve it is also possible to define a MoveTo parameter that can dictate a new filename to rename the retrieved file to. Additionally, the permission attributes of the retrieved file can also be set. A simplified IsExecutable flag can be used to make the file executable, but the more flexible Permissions field can be used to set the full permissions of the file. For these permissions the common four digit UNIX format is used.

XML definition

<planElement id="element id" name="element name" type="FileTransfer">
  <input ... />
  <output ... />
  <direction value="Store"/>
</planElement>
 
<planElement id="element id" name="element name" isexec="true" type="FileTransfer">
  <input ... />
  <output ... />
  <direction value="Retrieve"/>
  <mvto ... />
  <perms value="0700"/>
</planElement>

Conditional

The Conditional construct represents a set of flows that are only executed in case their respective conditions are met. The element follows the known pattern of if/else if/else construct. The if and else if flows are only executed in case their respective conditions are met. The else flow is only executed , if defined, if none of the if and else if flows have been executed. When the execution is started, the conditions of the flows are initialized and evaluated in turn until the first condition that is evaluated to true is found. The respective flow is executed. If none of the conditions are met, the else flow is executed if available. The conditions defined follow the syntax defined by Execution Conditions.

XML definition

<planElement id="element id" name="element name" type="Conditional">
  <if>
    <conditionalFlow>
      <conditionTree ... />
      <planElement ... />
    </conditionalFlow>
  </if>
  <elif>
    <conditionalFlow>
      <conditionTree ... />
      <planElement ... />
    </conditionalFlow>
    ...
  </elif>
  <else>
    <planElement ... />
  </else>
</planElement>

Loop

The Loop construct acts as a container of a sub plan which is executed iteratively until the loop condition defined is evaluated to false. If during the execution of the sub plan a break is caught, the loop is terminated without bubbling up the exception. The loop condition defined follows the syntax defined by Execution Conditions.

XML definition

<planElement id="element id" name="element name" type="Loop">
  <conditionTree ... />
  <planElement ... />
</planElement>

Break

The Break construct is simply used to be able to declare the termination of the execution before all its steps are completed or to break from a loop present somewhere higher in the hierarchy.

XML definition

<planElement id="element id" name="element name" type="Break">
  <message>...</message>
</planElement>

Wait

The Wait construct is simply used to pause the thread of execution that contains this element for a specific period of time. Although used by it self, the element can have some usability, it is expected to be used in conjunction with one of the condition enabled elements such as the ExecutionPlan_Elements#Loop element in such a way as to be able to define execution pausing until some condition is met.

XML definition

<planElement id="element id" name="element name" type="Wait">
  <wait value="..."/>
</planElement>

Try Catch Finally

The Try Catch Finally construct acts as the common programmatic try/catch/finally block of execution. During execution the sub element that is declared in the try block is executed. In case an exception is caught, the catch blocks are iterated to match the exception caught with one of the handled ones. If some catch block can handle the exception, the sub element defined in that block is executed. If the selected catch block does not define that the exception is re-thrown after the execution of the compensation code, the execution is moved to the finally block sub element. If however the exception is to be re-thrown or a new exception is thrown from the compensation code, the exception is bubbled up after the finally block is executed.

The matching of the error caught with the catch blocks is not performed as in typical programming languages such as for example Java. Error may be originating from some remote host where the original exception bubbled up to this construct was of some application specific type which is not available in the host evaluating this element. Therefore the ExecutionEngine cannot rely on strongly typed classes or types. To circumvent this problem, the matching is performed using the exception name. Either the fully qualified name or the simple name to have a more relaxed approach to error matching. A catch block may also specify an empty error name to act as a catch all.

XML definition

<planElement id="element id" name="element name" type="TryCatchFinally">
  <try>
    <planElement ... />
  </try>
  <catch>
    <catch name="java.lang.SecurityException" isFullName="true" rethrow="true">
      <planElement ... />
    </catch>
    <catch rethrow="false">
      <planElement ... />
    </catch>
  </catch>
  <finally>
    <planElement ... />
  </finally>
</planElement>

Checkpoint

The Checkpoint construct is meant to be used in order to create a restoration point for the data used in the context of an execution plan. These currently include only a set of Proxy Locators that are expected as inputs by the construct. The Record Store is used to create a persistent copy of the set of inputs. Additional configuration parameters needed for the Record Store initialization are also part of the element's definition.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers.

XML definition

<planElement id="element id" name="element name" type="Checkpoint">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <output ... />
  <inputs>
    ...
  </inputs>
  <checkpointConfig bufferedEntries="..." depthType="OnRequest" emptyBufferThresholdFactor="..." 
      inactiveGracePeriodInMilliseconds="..." leasingAccessCount="..." millisToWaitBeforeTimeout="..." multiplexType="..." typeOfLocator="...">
    <proxyConfig authenticate="..." encrypt="..." protocol="..." random="...">
      <ports>
        <range start="..." end="..."/>
        ...
      </ports>
    </proxyConfig>
    <storeConfig random="true">
      <ports>
        <range start="..." end="..."/>
        ...
      <ports>
    </storeConfig>
  </checkpointConfig>
</planElement>

Processing

The Processing construct acts as a processing step not directly relevant to any execution step. Its purpose is to enable "offline" processing of Variables outside any other plan element as they would be normally declared. The Parameters that are used to retrieve inputs or store outputs in other plan elements are usually used for on the fly processing of the inputs and outputs. Some values however may be needed more than once for multiple elements. Some parameters may also have complex definitions and there may be an interest in simplifying the definition of a construct.

The Processing construct is simply a placeholder for a number of Parameters. During execution, the list of Parameters are iterated and for each one the processing they define is invoked. The return values are not used, so to be of value, each parameter registered with this element should be of filtering type with an underlying filter that can store its output.

XML definition

<planElement id="element id" name="element name" type="Filter">
  <list>
    <param .../>
    <param .../>
    <param .../>
    ...
  </list>
</planElement>

Boundary

The Boundary construct represents a requirement that the sub plan that is defined under it is to be executed in a remote execution container. The access information for this remote container is specifically provided at definition time. During execution, the tree of execution under this element is moved to the remote execution container. Additionally to the sub plan moved to the remote host, the subset of the available Variables that are needed by this sub plan are also moved. Additionally, files locally available to the hosting node that executes the initiating end of the transfer, can be moved as attachments to the newly instantiated node using the same connection. The execution in the remote container can be isolated so as any byproducts of the execution do not overlap with other executions that take place in parallel. The isolation directory is automatically generated if requested Files that should be cleaned up after execution can also be declared.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers. This means that in case of error a different execution container can be picked and the execution restarted there.

After the execution in the remote execution container is finished, the Variables that are modified by the sub plan are send back to the caller and are used to update the local copy. If the execution terminated with an error, the error is retrieved and re-thrown.

XML definition

<planElement id="element id" name="element name" type="Boundary">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <boundaryConfig hostname="..." port="...">
    <nozzleConfig broadcast="..." restrict="..." type="..."/>
  </boundaryConfig>
  <isolation cleanUp="..." isolate="...">
    <baseDir>...</baseDir>
  </isolation>
  <cleanup>
    <file name="..."/>
    ...
  </cleanup>
  <attachments>
    <attachment type="..." cleanup="...">
      <value>...</value>
      <restore>...</restore>
      <permissions value="..."/>
    </attachment>
    ...
  </attachments>
</planElement>

Shell Script

The Shel Script construct acts as an invocation wrapper of a shell script, or in general an external executable, enabling it to be directly included in the execution of a plan. The executable is defined using its absolute or relative path located in the execution container that is hosting the execution of the plan element. Any parameters it may require to be provided are specified in the arguments collection of the element. Additionally, any environmental variables that need to be present for the execution of the command can also be defined and set prior to the execution. The execution of the external executable is done using a ProcessBuilder.

It is also possible to define the data that will be written to the process's standard input. The standard output and standard error of the process can be retrieved by setting the respective parameters. These inputs and outputs can be retrieved / dumped from /to files by setting respective flags.

The exit code of the executable can be used to identify successful completion. A mapping of the possible exit values, or a negation of them can be used to identify specific errors or successful completion. Depending on the exit code, the element can be defined to throw specific exception to signal different types of errors thereby binding non Java error to the internal error handling facilities.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers.

XML definition

<planElement id="element id" name="element name" type="Checkpoint">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <command>...</command>
  <arguments>
    <attrParam isFile="...">
      <param ... />
    </attrParam>
    ...
  </arguments>
  <stdIn isFile="...">
      <param ... />
  </stdIn>
  <stdOut isFile="...">
      <param ... />
  </stdOut>
  <stdErr isFile="...">
      <param ... />
  </stdErr>
  <stdExit>
      <param ... />
  </stdExit>
  <exitCodeErrors>
    <exitCodeError>
      <exitCode value="..." type="..."/>
      <errorFullName value="java.lang.Exception"/>
      <errorSimpleName value="Exception"/>
      <message value="..."/>
    </exitCodeError>
    ...
  </exitCodeErrors>
  <env>
    <item key="...">...</item>
    ...
  </env>
</planElement>

Java Object (POJO)

Web Service

Retrieved from "https://wiki.gcube-system.org/index.php?title=ExecutionPlan_Elements&oldid=8700"