WorkflowJDLAdaptor

Overview

This adaptor as part of the adaptors offered by the WorkflowEngine constructs an ExecutionPlan based on the description of a job defined in JDL syntax. This description can be of a single job or it can include a Directed Acyclic Graph (DAG) of jobs. The JDL description is parsed and the adaptor then processed the retrieved parsed info to create the ExecutionPlan. During the parsing procedure, the jobs Input and Output sandboxes are examined to determin what is the overall input and output set of the workflow. The input set is covered by the submitting client through the attached resources he provides. The output resources of the workflow are constructed from the elements found in all the jobs Output Sandboxes.

JDL attributes

DAG

Supported attributes in a JDL description defining a DAG of jobs are the following:

• Max_Running_Nodes (optional)

  The maximum number of concurrently running execution units. In cases where node collocation is not requested, this is performed at the level of Boundary element, while in cases of node collocation, since a single Boundary element is used, it is applied to the Shell plan element. More details on execution restrictions can be found in ExecutionEngine_Configuration.

• Nodes (mandatory)

  The Nodes attributes is where the jobs of a DAG are defined. At least one node must be defined.

• Dependencies (optional)

  The Dependencies attribute contains a list of pairs where the dependencies that formulate the DAG are defined. Only pairs of dependencies are supported and not inner lists. A pair in the form of {A,B} means that node B cannot be executed before node A has been completed.

• NodesCollocation (optional)

  Boolean indication on a requirement to host all DAG nodes in the same host.

• Rank (optional)

  In cases where node collocation is requested, this defines the rank function to use for the selection of the common host.

• Requirements (optional)

  In cases where node collocation is requested, this defines the requirements function to use for the selection of the common host.

Workflow Engine Extensions to the JDL syntax

• ParsingMode (optional)

  Whether the adaptor should create a plan that orchestrates the execution as a series of sequential and parallel steps, or by defining a bag of execution units with execution conditions made up from their dependency lists. Supported values for this are Plan and Bag for the two alternatives respectively.

• ConnectionMode (optional)

  When contacting a remote execution unit, the execution engine can either maintain a steady connection with the remote site or wait for the remote side to perform a callback to report its completion. Supported values for this are KeepAlive and Callback for the two alternatives respectively.

Job

Supported attributes in a JDL description defining a job are the following:

• JobType (mandatory)

  The type of the job. Currently only Normal job type is supported.

• Executable (mandatory)

  The executable that performs the actual job.

• Arguments (optional)

  The arguments to pass to the executable. The arguments that are included in this attribute are split using the space character as a delimiter. This means that no space containing phrase can be passed as a single argument.

• StdInput (optional)

  What to write to the executable's standard input once the execution is started. The content of the attribute is expected to be a file that either exists in the host initially, or is transfered as one of the elements contained in the Input Sandbox of the job.

• StdOutput (optional)

  Where to write the executable's standard output once the execution is started. The content of the attribute is expected to be a file that if needed to be later available will be included in the job's Output Sandbox.

• StdError (optional)

  Where to write the executable's standard error once the execution is started. The content of the attribute is expected to be a file that if needed to be later available will be included in the job's Output Sandbox.

• InputSandbox (optional)

  This element contains the files that need to be staged to the host that is executing the job.

• OutputSandbox (optional)

  This element contains the files that are produced by the job and need to be retrievable.

• Environment (optional)

  This element contains key value pairs that define the environmental variables and their value3s that need to be set for the execution of the job.

• RetryCount (optional)

  The number of retries that are to be performed in case a job fails once the execution is at the stage of executing the actual executable.

• ShallowRetryCount (optional)

  The number of retries that are to be performed in case there is a problem contacting the host to be used.

• Rank (optional)

  This element defines the rank function to use for the selection of the job host.

• Requirements (optional)

  This element defines the requirements function to use for the selection of the job host.

Workflow Engine Extensions to the JDL syntax

• ParsingMode (optional)

  Whether the adaptor should create a plan that orchestrates the execution as a series of sequential and parallel steps, or by defining a bag of execution units with execution conditions made up from their dependency lists. Supported values for this are Plan and Bag for the two alternatives respectively.

• ConnectionMode (optional)

  When contacting a remote execution unit, the execution engine can either maintain a steady connection with the remote site or wait for the remote side to perform a callback to report its completion. Supported values for this are KeepAlive and Callback for the two alternatives respectively.

Highlights

Parellization factor

Depending on the configuration, the adaptor will create the ExecutionPlan that will orchestrate the execution of a DAG of jobs either as a series of ppSequencePlanElement]] and FlowPlanElement elements or as a single BagPlanElement. The first case allows for a well defined series of operation but since the creation of such a series of constructs is an exercise on graph topological sorting, which as a problem can provide multiple answers that depending on the nature of the original graph might restrict the parallelization factor of the overall DAG, in cases of complex graphs, this case can damage the parallelization capabilities of the overall plan. The second case is much more dynamic. It allows for execution time decision making of the nodes to be executed. This of course comes as a tradeoff with increased complexity at runtime with respect to the well defined plan, but it can provide optimal parallelization capabilities.

Staging

Staging of input for the executables constituting the execution units of the jobs, is performed at a level of Input Sandbox defined for each job. The resources that are attached to the adaptor are stored in the StorageSystem and are retrieved in the node that hosts the Input Sandbox that declares them. The files declared in the Output Sandbox of a job are stored in the StorageSystem and information on the way to retrieve the output is provided through the Output Resources defined by the adaptor and are valid after the completion of the execution.

This means that staging of intermediate files is not performed because of the lack of a decrarative expression that can define the requirement. This means that is in the context of a DAG, node B is run after node A is completed, and node A produces an output that node B needs as input, there is no way of declaring this staging requirement using one of the existing JDL attributes (This point still needs validation as it may be covered by some JDL attribute that has not been incorporated in the supported ones). in later versions an extension attribute will be incorporated to support such staging.

Known limitations

• JDL parsing needs all declerations to start and begin with angle brackets ([,])
• JDL attributes parsing is case sensitive
• JDL Dependencies attribute only support pairs not inner lists
• Supported jobs are only those of type Normal
• Node collocation

  The case of node collocation in DAG jobs is not handled correctly because multiple BoundaryPlanElement are created. The node used is still a single one but it is contacted multiple times and data locality is not exploited correctly.

• The arguments defined for an executable in the respective JDL attribute, when passed to the ShellPlanElement are split using the space character (' ') as a delimiter. This way no space containing phrase can be passed as a single argument
• The Retry and Shallow Retry attributes of the JDl are treated equally and are used at the level of ShellPlanElement and not at the level of BoundaryPlanElement depending on the case
• After the execution completion no cleanup in the StorageSystem is done.
• Support staging of intermediate results