Difference between revisions of "IR Bootstrapper"

Revision as of 12:41, 14 October 2011

The IR Bootstrapper portlet provides a graphical user interface for executing sets of tasks on various resources of the infrastructure after the data import phase is completed. These tasks lead to the creation of other resources such as:

• Indexes
• Transformed metadata collections
• Open Search resources for open search collections

This portlet is based on a configuration file that is saved as a generic resource on the IS. This file is at XML format and it defines:

• The available tasks that can be executed
• The available jobTypes that can be used. These JobTypes define a sequential and/or parallel task executions for a given type of input to a given output
• The available jobs which are of type of the available jobTypes and provide all the specific inputs for this type.
  • The jobs are the ones that are available for execution on the resources.
  • A job can be extended by another job to define a more restrict job to execute (i.e. to be defined for a collection with a given name)
  • The user can define a new job by using the portlet's graphical user interface

Job Execution

The first tab of the portlet is divided into 2 main panels. At the left panel there is a tree with all the available collections. Clicking on a collection you can see all the jobs that can be executed on this collection. You can select any of the jobs and see the execution tree at the right panel. When a lock icon appears at a task of the selected job it means that this task is already completed for the selected collection and thus it won't be executed again. In order to execute this job you have to click on the button located on the top of the tree or you can check the checkbox and click on the button. This button is enabled when at least one job is checked.

Jobs Batch Submission

When you check more than one collections of the same job type, you can submit these jobs for batch execution. If these jobs require any extra user input at runtime a window appears asking for the extra input. The same input will be used for all the jobs that will be submitted using the batch mode.

When a job is submitted it is added at the Submitted jobs tree. You can go at the Submitted Jobs panel to check the state of each job.

• On each task an icon declares the current state: Running, Completed, Completed with warnings, Failed or Fulfilled Task
• You can see the execution log of each task by clicking on the '+' button.
• For each job you can abort the execution or you can remove it from the list

Job Designer

The second tab shows a tree with all the job types and all the defined jobs for each type. You can delete an existing job, display the execution tree of a job and/or clone an existing job to a new one
These changes update the bootstrapper's portlet configuration generic resource.

You can also create a new job using the graphical interface

• The job should have a name and be of a specific type
• For all the assignments of the specified type a value should be provided
  

Form more information about the jobs and jobTypes please refer to the section below

Bootstrapper Static Configuration

A static configuration in an XML format is required by the portlet in order to be initialized. This configuration is saved as a generic resource to the system's Information System. The configuration is created by the administrator when the portlet is released and can be enhanced later on using the portlet's Job Editor.

The configuration is consisted of 2 main parts:

• Types
• Jobs

Types

There are 3 different types that should be declared:

• Type: It is added by the administrator when the resource is created and declares the classes defined in the portlet's source code
  

In the current implementation of the portlet the Data types that are defined are the following:

<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.GCUBECollectionDataType" name="GCUBECollection" />          
 
<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.MetadataCollectionDataType" name="MetadataCollection" />
 
<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.OpenSearchDataType" name="OpenSearch" />
 
<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.ForwardIndexDataType" name="ForwardIndex" />
 
<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.GeoIndexDataType" name="GeoIndex" />
 
<type class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.data.FullTextIndexDataType" name="FullTextIndex" />

• TaskType: It is added by the administrator when the resource is created and declares the tasks that can be executed using the portlet. For each task type the input and the output should be defined and the allowed values can be one of the available Data types described above.
  

In the current implementation of the portlet the tasks that can be executed are the following:

<tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.FullTextIndexGenerationTaskType" name="FullTextIndexGenerationTask">
 
  <input type="GCUBECollection" />
 
  <output type="FullTextIndex" />
 
  <run>true</run>
 
</tasktype>
 
<tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.MetadataCollectionTransformationTaskType" name="MetadataCollectionTransformationTask">
 
  <input type="MetadataCollection" />
 
  <output type="MetadataCollection" />
 
  <run>true</run>
 
</tasktype>
 
<tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.GeoIndexGenerationTaskType" name="GeoIndexGenerationTask">
 
  <input type="MetadataCollection" />
 
  <output type="GeoIndex" />
 
  <run>true</run>
 
</tasktype>
 
<tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.OpenSearchGenerationTaskType" name="OpenSearchGenerationTaskType">
 
  <input type="GCUBECollection" />
 
  <output type="OpenSearch" />
 
  <run>true</run>
 
</tasktype>
 
<tasktype class="org.gcube.portlets.admin.irbootstrapperportlet.gwt.server.types.task.ForwardIndexGenerationTaskType" name="ForwardIndexGenerationTask">
 
  <input type="GCUBECollection" />
 
  <output type="ForwardIndex" />
 
  <run>true</run>
 
</tasktype>

• JobType: The JobTypes are added by the administrator and declare a set of tasks that will be executed and how this execution will be executed. You can declare either parallel or sequential sets of assignments.

<jobtype description="Creates the required fulltext indices for a collection." name="FTIndexCollection">
 
    <input type="GCUBECollection" />
 
    <ChainExecution>
 
       <ChainConnectionAssignments>
 
          <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IdOfIndexManagerToAppend" value="%Create_MC_ft_index.output.IndexID" />
 
       </ChainConnectionAssignments>
 
     </ChainExecution>
 
     <jobDefinition>
 
        <parallel>
 
          <sequential>
 
             <assign to="%Create_MC_ft_index.input" value="%FTIndexCollection.input" />
 
             <assign to="%Create_MC_ft_index.output.IndexedCollectionID" value="%Create_MC_ft_index.input.ColID" />
 
             <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.NumberOfLookups" value="2" />
 
             <task name="Create_MC_ft_index" tasktype="FullTextIndexGenerationTask" />
 
           </sequential>
 
         </parallel>
 
     </jobDefinition>
 
</jobtype>

In this example the JobType creates FullText indexes. It takes as input a GCUBECollection and it will execute the FullTextIndexGenerationTask taskType that should be already defined in the configuration. It performs the required assignments for this task by providing the desired input and output. Furthermore it declares that this task should create 2 index lookups. Notice here that these assignments will be run sequentially as defined in the XML.

• In order to declare that jobs of a specific JobType can be executed in a batch mode and share the output of the previous executed job you should add this element to the JobType definition:

    <ChainExecution>
 
       <ChainConnectionAssignments>
 
          <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IdOfIndexManagerToAppend" value="%Create_MC_ft_index.output.IndexID" />
 
       </ChainConnectionAssignments>
 
     </ChainExecution>

In this assignment you declare that in a batch mode execution the specific assignment will take as value the specific output value of the previous job (i.e. the IndexID of a Full Text Index will be used as the Index Manager ID of the next Job. This will force both indexes to be created under the same WS-resource Index Manager).

Jobs

You can define as many jobs you want by using the portlet's visual Job Editor/Creator. The portlet helps the creation by suggesting the JobTypes and the assignments.
The type of each job should be one of the declared JobTypes and based on the type every job should declare the needed assignments. You can see the XML of job for the FTIndexCollection JobType below:

<job jobtype="FTIndexCollection" name="FTIndexSpeciesMapsCollection">
 
   <initialization>
 
     <assign to="%FTIndexCollection.input.ColName" value="FARM AquaMaps: Species Maps" />
 
      <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IndexTypeID" value="ft_2.0" />
 
      <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.TransformationXSLTID" value="$BrokerXSLT_wrapperFT" />
 
      <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.Views" value="[ bc7b7352-1471-4631-ad28-50b4a6f92dfb, 229716ce-c226-4390-9ef1-e54936664d59 ]" />
 
      <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.ViewsXSLTSIDs" value="[ $BrokerXSLT_aquamaps_anylanguage_to_ftRowset_anylanguage, $BrokerXSLT_FARM_dc_anylanguage_to_ftRowset_anylanguage ]" />
 
      <assign to="%Create_MC_ft_index.FullTextIndexGenerationTask.IdOfIndexManagerToAppend" userInputLabel="ID of index management resource to append" value="%userInput" />
 
    </initialization>
 
</job>

In this example the job has the name: FTIndexSpeciesMapsCollection and it declares through the assignments that it can only be matched for the GCUBECollection with name: "FARM AquaMaps: Species Maps" and it also defines all the other needed assignments needed for the Full Text Index creation.

• It is worth noticing that if you want a job to ask for a user input at runtime then you should declare at the respective assignment the specific value: %userInput and the attribute: userInputLabel should contain a description of the type of value you are expecting to be provided by the user