Difference between revisions of "WorkflowGridAdaptor"
(→Processing filters) |
|||
Line 76: | Line 76: | ||
</pre> | </pre> | ||
The filter scans through the output and parses it into a series of records/events. Every event extracted is send back to the caller utilizing the [[ExecutionEngine_EventFrameWork]]. | The filter scans through the output and parses it into a series of records/events. Every event extracted is send back to the caller utilizing the [[ExecutionEngine_EventFrameWork]]. | ||
+ | |||
+ | =Usage= | ||
+ | The following snippets demonstrate the usage of the adaptor. | ||
=Known limitations= | =Known limitations= | ||
Line 91: | Line 94: | ||
*Cleanup files stored in [[StorageSystem]] as intermediate steps after completion | *Cleanup files stored in [[StorageSystem]] as intermediate steps after completion | ||
*Allow user cancellation | *Allow user cancellation | ||
+ | *There is currently no web service interface to access the adaptor. The creation of the plan is done through command line utilities. |
Revision as of 18:21, 29 January 2010
Contents
Overview
This adaptor constructs an ExecutionPlan that can mediate to submit a job described through a JDL file using a gLite Grid UI node. After its submission the job is monitored for its status and once completed the output files are retrieved and stored in the StorageSystem. The resources that are provided and need to be moved to the Grid UI are all transfered through the StorageSystem. They are stored once the plan is constructed and are then retrieved once the execution is started. This does not include the provided user proxy which is transfered as an attachment directly to the remote node to allow secure transfer if the SSL communication option is enabled.
Plan Template
The entire execution process takes place in the gLite Grid UI node. This node is picked from the InformationSystem and is currently chosen randomly from all the available ones. Currently once the node has been picked, the execution cannot be moved to a different one even if there is a problem communicating with that node. The execution that takes place is a series of steps executed sequentially. These steps include the following:
- Contact the remote node
- Retrieval of the data stored in the StorageSystem and these include the resources marked as Configuration, Input Data, and JDL description
- Submit the job using the provided JDL file and optionally any configuration additionally provided using the provided user proxy certificate
- Go into a loop until either the job is completed or a timeout has expired (If a timeout has been set)
- Wait for a defined period
- Retrieve the job status
- Retrieve the job logging info
- Process the results of the above two steps
- Check the reason the loop ended
- If a timeout happened, cancel the job
- Otherwise retrieve the output files of the job
Highlights
Execution timeout
A timeoput can be set so that if after some period the execution is not completed, the job is canceled and the execution aborted
Polling interval
After the job is submitted, in regular intervals the status as well as logging info associated with the job is polled to decide on the way to continue the workflow and to report back on the progress. This interval is configurable by the client.
Processing filters
Status filter
Taking advantage the extensible filtering capabilities of the ExecutionEngine_FilterFramework the WorkflowEngine defines an external filter that can process the output of the glite-wms-job-status command. This way, outputs of the following format can be parsed and the status of the job can be extracted and recognized as one of the known statuses (Submitted, Waiting, Ready, Scheduled, Running, Done, Cleared, Aborted, Canceled)
************************************************************* BOOKKEEPING INFORMATION: Status info for the Job : https://some_host:some_port/some_identifier Current Status: Done (Success) Logged Reason(s): - Job got an error while in the CondorG queue. - Job terminated successfully Exit code: 0 Status Reason: Job terminated successfully Destination: some_host:som_port/some_job_manager Submitted: some_date *************************************************************
Logging info filter
Taking advantage the extensible filtering capabilities of the ExecutionEngine_FilterFramework the WorkflowEngine defines an external filter that can process the output of the glite-wms-job-logging-info command. This way, outputs of the following format can be parsed
********************************************************************** LOGGING INFORMATION: Printing info for the Job: https://som_host:som_port/some_identifier - - - Event: RegJob - source = UserInterface - timestamp = Fri Feb 20 10:30:16 2004 - - - Event: Transfer - destination = NetworkServer - result = START - source = UserInterface - timestamp = Fri Feb 20 10:30:16 2004 - - - Event: Transfer - destination = NetworkServer - result = OK - source = UserInterface - timestamp = Fri Feb 20 10:30:19 2004 - - - Event: Accepted - source = NetworkServer - timestamp = Fri Feb 20 10:29:17 2004 - - - Event: EnQueued - result = OK - source = NetworkServer - timestamp = Fri Feb 20 10:29:18 2004 [...] **********************************************************************
The filter scans through the output and parses it into a series of records/events. Every event extracted is send back to the caller utilizing the ExecutionEngine_EventFrameWork.
Usage
The following snippets demonstrate the usage of the adaptor.
Known limitations
Some of the know limitations of the currently created plan are listed below. This limitations are mainly because of simplicity of the plan and not because off the lack of constructs to cover them. This list will be updated in later versions of the adaptor that will enrich the produced plan.
- gLite Grid UI node selection
- Have a more elaborate selection strategy for the node submitting the jobs
- Relocation of execution
- Once the UI node has been picked it cannot be moved. This means that if after picking it the node becomes unreachable, the whole workflow is lost. Allow relocation, cancelation of proevious submittion and resubmission, continue monitoring of previously submitted job, etc
- Error Handling
- Now all errors are fatal. Be more resilient when errors are non critical
- Results retrieval
- Only retrieve output files if completion is successful
- Add SSL option in communication
- Allow multiple JDLs and collection style submission
- Cleanup files stored in StorageSystem as intermediate steps after completion
- Allow user cancellation
- There is currently no web service interface to access the adaptor. The creation of the plan is done through command line utilities.