ExecutionPlan Elements

From Gcube Wiki
Jump to: navigation, search

Overview

All plan elements are automatically assigned with an unique identifier, through which they are retrievable and identified. Additionally, all elements can be assigned with a name that they use when emitting progress events. This way the execution monitoring can be made more user friendly.

The elements listed below can be roughly separated in two large categories, based on the position they may have in the execution tree.

All the Nodes elements have placeholders where other elements, either Nodes themselves or Leaves can be plugged in. The Leaves on the other hand can be plugged in other Nodes but the execution plan path ends to them.

All elements, while executing report on their progress, use the Events mechanism to inform the client on its state. For the elements that perform operations internal to the engine, this keeps a constant event flow reporting back to the client. The elements though that pass the execution to external elements, such as the Web Service, Java Object (POJO) and Shell Script, when the execution is forwarded to these elements, this flow of information is interrupted. To cover this gap, the Execution Context construct is provided.

Plan Elements

Sequence

The Sequence construct offers a simple ordering utility for a set of execution steps. This element is a simple container maintaining the order of execution of its sub elements. When executed, the ordered list of sub elements is executed one after the other, and the Sequence element is completed when its last sub-element is completed.

XML definition 

<planElement id="element id" name="element name" type="Sequence">
  <list>
    <planElement .../>
    <planElement .../>
    <planElement .../>
    ...
  </list>
</planElement>

Flow

The Flow construct offers a simple parallelization utility for a set of execution steps. This element is a simple container, maintaining a set of sub elements that can be executed in parallel. When executed, the set of sub elements are executed all at once in parallel, and the Flow is completed when all the sub elements that have started are terminated. If one of the elements is terminated with an error, after all the elements have terminated, the exception indicating the error is re-thrown. If more than one elements finished with an error, only the first error found iterating through the set of sub elements is thrown.

XML definition 

<planElement id="element id" name="element name" type="Flow">
  <list>
    <planElement .../>
    <planElement .../>
    <planElement .../>
    ...
  </list>
</planElement>

Bag

The Bag construct acts as a container of other sub elements. It orchestrates their execution, based on their execution preconditions. These elements, as well as their conditions are evaluated every time one of the previously running elements have completed their execution. While this element is executed, every time an element previously running has terminated, the set of not yet run elements is scanned. If no element can be run while no more elements are still running, the process is terminated, if such a behavior is requested. Even if more elements are still running, if termination condition is set and is evaluated to true, the execution of the element is terminated. This condition follows the syntax defined by Execution Conditions. The execution preconditions of each element also follow the syntax defined by Execution Conditions.

XML definition 

<planElement id="element id" name="element name" type="Bag">
  <list>
    <bagElement>
      <planElement ... />
      <conditionTree .../>
    </bagElement>
    <bagElement>
      <planElement ... />
      <conditionTree .../>
    </bagElement>
    ...
  </list>
  <termination onNoProgress="true">
    <conditionTree .../>
  </termination>
</planElement>

Know issues

It is possible that, if it has not been requested for the Bag to terminate when no progress can be made, and the termination condition, if set, does not cover this case, that the Bag element will never terminate its execution.

Furthermore, currently the execution conditions of the bag's elements are only evaluated once at the beginning of the execution and then every time a previously running element terminates. This means that, even if some execution condition is not dependent of some other element being terminated, it will not be discovered before that.

File Transfer

The File Tranfer construct can be used to mediate at the execution level the staging of files to and from the StorageSystem. The transfer can have a direction of either Store or Retrieve. In case the direction is Store, the File Tranfer utility expects as input the file path that is to be stored in the StorageSystem and is located in the same hosting machine as the one executing the element. After the operation is completed, the output of the File Transfer will be the identifier assigned to the stored file by the StorageSystem. In case the direction is Retrieve, the input is expected to be the identifier of the document that is stored in the StorageSystem and after the execution, the output of Filte Transfer will have the local filename where the document is stored.

When the File Transfer direction is Retrieve it is also possible to define a MoveTo parameter that can dictate a new filename to rename the retrieved file to. Additionally, the permission attributes of the retrieved file can also be set. A simplified IsExecutable flag can be used to make the file executable, but the more flexible Permissions field can be used to set the full permissions of the file. For these permissions the common four digit UNIX format is used.

XML definition 

<planElement id="element id" name="element name" type="FileTransfer">
  <input ... />
  <output ... />
  <direction value="Store"/>
</planElement>
 
<planElement id="element id" name="element name" isexec="true" type="FileTransfer">
  <input ... />
  <output ... />
  <direction value="Retrieve"/>
  <mvto ... />
  <perms value="0700"/>
</planElement>

Conditional

The Conditional construct represents a set of flows that are only executed in case their respective conditions are met. The element follows the known pattern of if/else if/else construct. The if and else if flows are only executed in case their respective conditions are met. The else flow is only executed , if defined, if none of the if and else if flows have been executed. When the execution is started, the conditions of the flows are initialized and evaluated in turn until the first condition that is evaluated to true is found. The respective flow is executed. If none of the conditions are met, the else flow is executed if available. The conditions defined follow the syntax defined by Execution Conditions.

XML definition 

<planElement id="element id" name="element name" type="Conditional">
  <if>
    <conditionalFlow>
      <conditionTree ... />
      <planElement ... />
    </conditionalFlow>
  </if>
  <elif>
    <conditionalFlow>
      <conditionTree ... />
      <planElement ... />
    </conditionalFlow>
    ...
  </elif>
  <else>
    <planElement ... />
  </else>
</planElement>

Loop

The Loop construct acts as a container of a sub plan which is executed iteratively until the loop condition defined is evaluated to false. If during the execution of the sub plan a break is caught, the loop is terminated without bubbling up the exception. The loop condition defined follows the syntax defined by Execution Conditions.

XML definition 

<planElement id="element id" name="element name" type="Loop">
  <conditionTree ... />
  <planElement ... />
</planElement>

Break

The Break construct is simply used to be able to declare the termination of the execution before all its steps are completed or to break from a loop present somewhere higher in the hierarchy.

XML definition 

<planElement id="element id" name="element name" type="Break">
  <message>...</message>
</planElement>

Wait

The Wait construct is simply used to pause the thread of execution that contains this element for a specific period of time. Although used by it self, the element can have some usability, it is expected to be used in conjunction with one of the condition enabled elements such as the Loop element in such a way as to be able to define execution pausing until some condition is met.

XML definition 

<planElement id="element id" name="element name" type="Wait">
  <wait value="..."/>
</planElement>

Try Catch Finally

The Try Catch Finally construct acts as the common programmatic try/catch/finally block of execution. During execution the sub element that is declared in the try block is executed. In case an exception is caught, the catch blocks are iterated to match the exception caught with one of the handled ones. If some catch block can handle the exception, the sub element defined in that block is executed. If the selected catch block does not define that the exception is re-thrown after the execution of the compensation code, the execution is moved to the finally block sub element. If however the exception is to be re-thrown or a new exception is thrown from the compensation code, the exception is bubbled up after the finally block is executed.

The matching of the error caught with the catch blocks is not performed as in typical programming languages such as for example Java. Error may be originating from some remote host where the original exception bubbled up to this construct was of some application specific type which is not available in the host evaluating this element. Therefore the ExecutionEngine cannot rely on strongly typed classes or types. To circumvent this problem, the matching is performed using the exception name. Either the fully qualified name or the simple name to have a more relaxed approach to error matching. A catch block may also specify an empty error name to act as a catch all.

XML definition 

<planElement id="element id" name="element name" type="TryCatchFinally">
  <try>
    <planElement ... />
  </try>
  <catch>
    <catch name="java.lang.SecurityException" isFullName="true" rethrow="true">
      <planElement ... />
    </catch>
    <catch rethrow="false">
      <planElement ... />
    </catch>
  </catch>
  <finally>
    <planElement ... />
  </finally>
</planElement>

Checkpoint

The Checkpoint construct is meant to be used in order to create a restoration point for the data used in the context of an execution plan. These currently include only a set of Proxy Locators that are expected as inputs by the construct. The Record Store is used to create a persistent copy of the set of inputs. Additional configuration parameters needed for the Record Store initialization are also part of the element's definition.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers.

XML definition 

<planElement id="element id" name="element name" type="Checkpoint">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <output ... />
  <inputs>
    ...
  </inputs>
  <checkpointConfig bufferedEntries="..." depthType="OnRequest" emptyBufferThresholdFactor="..." 
      inactiveGracePeriodInMilliseconds="..." leasingAccessCount="..." millisToWaitBeforeTimeout="..." multiplexType="..." typeOfLocator="...">
    <proxyConfig authenticate="..." encrypt="..." protocol="..." random="...">
      <ports>
        <range start="..." end="..."/>
        ...
      </ports>
    </proxyConfig>
    <storeConfig random="true">
      <ports>
        <range start="..." end="..."/>
        ...
      <ports>
    </storeConfig>
  </checkpointConfig>
</planElement>

Processing

The Processing construct acts as a processing step not directly relevant to any execution step. Its purpose is to enable "offline" processing of Variables outside any other plan element as they would be normally declared. The Parameters that are used to retrieve inputs or store outputs in other plan elements are usually used for on the fly processing of the inputs and outputs. Some values however may be needed more than once for multiple elements. Some parameters may also have complex definitions and there may be an interest in simplifying the definition of a construct.

The Processing construct is simply a placeholder for a number of Parameters. During execution, the list of Parameters are iterated and for each one the processing they define is invoked. The return values are not used, so to be of value, each parameter registered with this element should be of filtering type with an underlying filter that can store its output.

XML definition 

<planElement id="element id" name="element name" type="Filter">
  <list>
    <param .../>
    <param .../>
    <param .../>
    ...
  </list>
</planElement>

Boundary

The Boundary construct represents a requirement that the sub plan that is defined under it is to be executed in a remote execution container. The access information for this remote container is specifically provided at definition time. During execution, the tree of execution under this element is moved to the remote execution container. Additionally to the sub plan moved to the remote host, the subset of the available Variables that are needed by this sub plan are also moved. Additionally, files locally available to the hosting node that executes the initiating end of the transfer, can be moved as attachments to the newly instantiated node using the same connection. The execution in the remote container can be isolated so as any byproducts of the execution do not overlap with other executions that take place in parallel. The isolation directory is automatically generated if requested Files that should be cleaned up after execution can also be declared.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers. This means that in case of error a different execution container can be picked and the execution restarted there.

After the execution in the remote execution container is finished, the Variables that are modified by the sub plan are send back to the caller and are used to update the local copy. If the execution terminated with an error, the error is retrieved and re-thrown.

XML definition 

<planElement id="element id" name="element name" type="Boundary">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <boundaryConfig hostname="..." port="...">
    <nozzleConfig broadcast="..." restrict="..." type="..."/>
  </boundaryConfig>
  <isolation cleanUp="..." isolate="...">
    <baseDir>...</baseDir>
  </isolation>
  <cleanup>
    <file name="..."/>
    ...
  </cleanup>
  <attachments>
    <attachment type="..." cleanup="...">
      <value>...</value>
      <restore>...</restore>
      <permissions value="..."/>
    </attachment>
    ...
  </attachments>
</planElement>

Shell Script

The Shel Script construct acts as an invocation wrapper of a shell script, or in general an external executable, enabling it to be directly included in the execution of a plan. The executable is defined using its absolute or relative path located in the execution container that is hosting the execution of the plan element. Any parameters it may require to be provided are specified in the arguments collection of the element. Additionally, any environmental variables that need to be present for the execution of the command can also be defined and set prior to the execution. The execution of the external executable is done using a ProcessBuilder.

It is also possible to define the data that will be written to the process's standard input. The standard output and standard error of the process can be retrieved by setting the respective parameters. These inputs and outputs can be retrieved / dumped from /to files by setting respective flags.

The exit code of the executable can be used to identify successful completion. A mapping of the possible exit values, or a negation of them can be used to identify specific errors or successful completion. Depending on the exit code, the element can be defined to throw specific exception to signal different types of errors thereby binding non Java error to the internal error handling facilities.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers.

XML definition 

<planElement id="element id" name="element name" type="Checkpoint">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <command>...</command>
  <arguments>
    <attrParam isFile="...">
      <param ... />
    </attrParam>
    ...
  </arguments>
  <stdIn isFile="...">
      <param ... />
  </stdIn>
  <stdOut isFile="...">
      <param ... />
  </stdOut>
  <stdErr isFile="...">
      <param ... />
  </stdErr>
  <stdExit>
      <param ... />
  </stdExit>
  <exitCodeErrors>
    <exitCodeError>
      <exitCode value="..." type="..."/>
      <errorFullName value="java.lang.Exception"/>
      <errorSimpleName value="Exception"/>
      <message value="..."/>
    </exitCodeError>
    ...
  </exitCodeErrors>
  <env>
    <item key="...">...</item>
    ...
  </env>
</planElement>

Java Object (POJO)

The Java Object construct acts as an invocation wrapper of a Java object enabling it to be directly included in the execution of a plan. The Java class that is to be used is specified using its full name and is accessed using reflection. The list of actual invocations on this class are specified declaratively.

The list of invocations is organized as an ordered list, and the invocations may be either constructor calls or method invocations. The method invocations can be made to both member functions as well as static methods. The values provided as arguments to the method invocations are retrieved from Parameters defined at the level of plan. Any return value of method invocation can also be stored as a Variable using a Parameter.

Every time an instance of the class is created because the invocation type of call is a constructor, if it is so requested and the instantiated object supports it, an Execution Context is created and provided to the instance.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers.

XML definition 

<planElement id="element id" name="element name" type="POJO">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <className value="..."/>
  <calls>
    <call type="..." order="...">
      <methodName value="..."/>
      <arguments>
        <argument type="..." order="..." name="...">
          <param ... />
        </argument>
        ...
      </arguments>
      <output>
        <param ... />
      </output>
    </call>
    ...
  </calls>
  <context supported="...">
    <contextConfig type="..." proxytype="..." keepAlive="...">
      <config random="..." authenticate="..." encrypt="..." protocol="...">
        <ports>
          <range start="..." end="..."/>
          ...
        <ports>
      </config>
    </contextConfig>
  </context>
</planElement>

Web Service

The Web Service construct acts as an invocation wrapper of a web service enabling it to be directly included in the execution of a plan. The web service that is to be used is specified using its end point. The list of actual invocations on this class are specified declaratively.

The list of invocations is organized as an ordered list. Every method invocation is performed based on the method name, and the action urn. A single argument is expected for each method which is the full SOAP envelop of the execution. This envelop is provided as a Parameter which means that for its production a number of processing steps can be made in order for it to be constructed. For each invocation, a number of post envelop creation Filters can also be defined. After the SOAP envelop is created, they are are used to further process the envelop before the invocation. Any return value of a method invocation can be stored as a Variable using a Parameter.

Every time an invocation to the web service is made, if it is so requested, an Execution Context is created and provided. This context is supplied as a member of the SOAP header and injected using the post envelop creation filters.

This construct also offers the possibility to react to errors that may happen during its execution using contingency reactions as these are described in Contingency Triggers. This means that in case of error a different end point can be picked and the execution restarted there.

XML definition 

<planElement id="element id" name="element name" type="WS">
  <triggers>
    <contingency .../>
    ...
  </triggers>
  <endpoint>...</endpoint>
  <calls>
    <call type="..." order="...">
      <actionURN>...</actionURN>
      <executionContextToken>...</executionContextToken>
      <methodName value="..."/>
      <arguments>
        <argument type="..." order="..." name="...">
          <param ... />
        </argument>
      </arguments>
      <output>
        <param ... />
      </output>
      <postFilters>
        <filter ... />
        ...
      </postFilters>
    </call>
    ...
  </calls>
  <context supported="...">
    <contextConfig type="..." proxytype="..." keepAlive="...">
      <tcpProxyConfig random="..." authenticate="..." encrypt="..." protocol="...">
        <ports>
          <range start="..." end="..."/>
          ...
        <ports>
      </tcpProxyConfig>
      <nozzleConfig type="..." broadcast="..." restrict="..."/>
    </contextConfig>
  </context>
</planElement>

Execution Conditions

One of the constructs that makes the Execution Plan versatile is the capability of defining run time conditions which are evaluated against run time values. A condition is a boolean decision tree. This tree supports two types of nodes

  • Nodes
    A node can have a number of children which can be either other Nodes or Leaves. These children are combined with a verb which can be either a boolean AND operator or a boolean OR operator. There is also the possibility of post processing the result of the cumulative boolean operator by adding a negation over the result.
  • Leaves
    A leaf contains one of the supported conditions that are evaluated against a number of defined Parameters. The currently supported conditions are the following:
    • Boolean Value
      This condition simply checks the value retrieved by an input parameter which it expects to be of boolean value and returns this value.
    • Counter
      This condition once initialized starts a counter like behavior, and every time it is called to evaluate its condition, will increment this counter by some defined value (increasing or decreasing it). While this counter is within some defined boundaries, inclusive or not, it returns true. Once the counter has moved beyond these boundaries, the condition evaluates to false.
    • Range
      This condition checks if some parameter's value is within a defined range of values. If so, it returns true. Otherwise the condition evaluates to false
    • Timeout
      This condition once initialized marks the current time. Every time it is requested to evaluate its condition it will check if the time past since its initialization is within some defined period. If it is, true is returned. Otherwise false.
    • Value Available
      This condition checks if the defined Variable is marked as available, and has its value initialized or not. if it is, true is returned. Otherwise, false.

Most of the values that take part in the above conditions are accessed through Variables and Parameters. This way, the behavior and parametrization of these conditions are subject to dynamic configuration and they may in fact be instantiated from some previous step of the execution.

XML definition 

<conditionTree>
  <treeElement type="Node" verb="AND" post="AsIs">
    <treeElement type="Node" verb="OR" post="Negate">
      <treeElement type="Leaf">
        <condition type="Timeout">
          <threshold value="some timeout in milliseconds"/>
        </condition>
      </treeElement>
      <treeElement type="Leaf">
        <condition type="IsAvailable">
          <var name="variable name"/>
        </condition>
      </treeElement>
      <treeElement type="Leaf">
        <condition type="DecimalRange">
          <start>
            <param ... />
          </start>
          <end>
            <param ... />
          </end>
          <current>
            <param ... />
          </current>
          <leftinclusive value="true"/>
          <rightinclusive value="false"/>
        </condition>
      </treeElement>
    </treeElement>
    <treeElement type="Leaf">
      <condition type="BooleanVariable">
        <flag>
          <param .../>
        </flag>
      </condition>
    </treeElement>
    <treeElement type="Leaf">
      <condition type="Counter">
        <start>
          <param .../>
        </start>
        <end>
          <param .../>
        </end>
        <current>
          <param .../>
        </current>
        <increment>
          <param .../>
        </increment>
        <rightinclusive value="true"/>
      </condition>
    </treeElement>
  </treeElement>
</conditionTree>
Retrieved from "https://wiki.gcube-system.org/index.php?title=ExecutionPlan_Elements&oldid=12567"