Resource Broker

The Resource Broker represents an ad hoc gCube service promoting and supporting the optimal selection and usage of resources during the VRE deployment phase. In particular, it is invoked to select the most appropriate pool of gHNs to be used, among those available in the context of the VRE, during the deployment of the services needed to operate the VRE. Because of this, it interacts with:

• the Information System (IS) to be aware of the available gHNs as well as of their distinguishing features (e.g. the number of Running Instances currently hosted by it, the RAM the machine is equipped with) and
• with the Virtual Organisation Management (VO-Management) to act securely and filter out the gHNs that falls out of the operational context of the VRE. From an architectural point of view it mainly consists of a service implementing the matchmaking algorithm.

Deployment phase

Several steps are involved during the deployment of a VRE. The green circles are labeled with numbers representing the natural succession of actions. The labeled boxes are the actors by means of GHNs and the services instantiated on them that take part to the protocol (the other services published on such GHNs are omitted for a better readability). Finally the white box containing several GHNs depicts a typical situation where (a group of) generic GHNs are dedicated to host the freshly deployed VRE instances.

Figure 1. The complete VRE deployment phase.

The Figure 1 reports a scenario of VRE instantiation. Here, we briefly recall the main steps that are performed (we assume to start from a request of VRE instantiation coming from the portal interface and that VREa is the name to assign to it):

1. The portal requires the VRE Modeler to create a VRE (we call it VREa).
2. Internally the VRE Modeler compiles a list of mandatory services plus optional services (e.g. GeoSpatial Search, support for data annotation). Actually these information are statically bound into the DB.
3. The VRE Modeler accesses the VO Manager (a particular instance of VRE Manager defined at VO scope level) requesting to create an instance of the VRE Manager (Resource Manager of VREa) that will handle the requests for the resources for the scope of VREa.
   1. the VO Manager asks the Software Repository the list of packages needed to make the deploy (the ones required to instantiate the VRE Manager of VREa).
4. The VO Manager asks the Deployer on the gHN of VRE Manager (Resource Manager of VREa) to create the instance of VRE Manager (at VREa Scope).
   1. The Deployer of VREa gets from the Software Repository the required packages needed for deployment. Installs packets. Activates service instances (in this case the only VRE manager).
   2. The Deployer of VREa creates and activates the VRE Manager of VREa (now the gHN of VREa is ready - VREa is instantiated).
5. The VRE Modeler contacts the VREa Manager trough VRE Manager AddResources for allocating resources for VREa.
6. VREa Manager asks the list of packages needed to all the services of VREa (just instantiated).
7. VREa Manager distributes to the generic gHN the services to deploy and to run (services will be properly distributed among such nodes). A workflow similar to steps 4.1 and 4.2 will starts on the local deployers of generic gHN for getting the required gCube packages from the Software Repository.

Design

This Section will introduce the main concepts and the development strategies adopted at Resource Broker design phase.

Service architecture and code design

The whole project consists of several parts (reflected in the structure of packages) here summarized:

• configuration: various parameters characterizing the execution of BrokerService. Up to now they are grouped inside a single file. As future plan they are supposed to be partially moved in the jndi configuration file (so that they can be directly configured by end user). The access to all configurable properties will be demanded to an intermediate component responsible to retrieve and store them transparently to where they are located.
• contexts: a constraint for building Stateful WS-Resources in the gCube framework is that each service declares its own StatefulBrokerContext. Aside, the ServiceContext represents the context of each service instance and keeps no persistence. Both the contexts are managed by a ResourceBinder that properly instantiates and maintain them.
• resources: the SingletonResourceStorage is the singleton instance keeping the persistence of ResourceBroker status while the BrokerHome class is mandatory for storing internally the persistent resource and for retrieving it. For further details on how the state is mantained in gCube, refer to Building gCube Services Tutorial.
• types: contains the data structures acting as support for the ResourceBroker. Here we find:
  • CustomXPathQueries: a set of customized queries to the IS. Manly introduced to retrieve from the IS a compact description of GHNs searched, with the only information needed on this side (e.g. ID of the GHN and the number (a counter) of Running Instances actually on it).
  • GHNDescriptor: the local description of a GHN, with the subset of information needed to apply decision planning.
  • GHNReservation: each request to the ResourceBroker is coupled with an unique identifier and a reservation status where the reserved GHNs are stored. Each reservation has an expiration date and keeps track of all GHNs (their descriptors) assigned during the workflow planning. Once a reservation fails, or the expiration time elapses or a feedback is sent back, the reservation is released (and, accordingly, all the reserved GHNs unlocked).
  • SortableElement: generic container for sortable elements. Wraps generic elements inside it and is built up an index that is used to apply sorting algorithms. Sortable elements consist essentially of templates <T,V> where the first parameter (T) identifies the type for sortable indexes (e.g. int, long, ...) and the second parameter (V) constitutes the type of wrapped element.
  • Tuple: a general purpose definition of tuples. A tuple is a sequence (or ordered list) of finite length. Typical operations allowed on tuples are: union, append and equals (to compare them).
• services: the components implementing the core functionalities of ResourceBroker. Here are defined: GHNReservationHandler acting as proxy in behalf of the other services and the persistent resource maintaining the state of reservations. ISClientRequester that intermediates with the IS making the queries in behalf of Broker Service and returns the results encapsulated inside proper data structures defined in types package. BrokerService provides the implementation of the externally published web service allowing to request planning decisions during the VRE deployment phase.
• threads: the deployment plans are made in accordance to some parameters relative to the GHNs that are remotely retrieved from the IS. Once retrieved the set of GHNs active in a precise Scope, they are locally stored inside the persistent state of the service in order to avoid multiple requests to the IS for each request. For a such reason the locally stored information of GHNs must be refreshed at some intervals of time. TUpdateGHNProfiles is a timed thread that is responsible to update the profiles of GHNs defined in a given scope (or to remove from the local cache the ones no more available on the IS). TRevokeReservations is demanded to remove from the list of reserved GHNs the ones for which the time has expired. Both these threads are defined as subclasses of TimedThread, consisting of a scheduled thread that is frequently activated on a delay of time. It applies a passive wait during idle time. As support class here we find TimedThreadsStorage where these kinds of thread can be registered so that the exit of all of them can be directly required to this class.
• notifications: over the scheduled threads here, as additional support, we have two particular services that are subscribed on the IS-Registry for the notifications sent by IS-Notifier regarding modifications to the GHNs relevant at BrokerService Scope. The notification subscribers play a complementary role with respect to threads so that once the registry is not available or down for a while threads supply its functionality and, conversely, if the notifications arrive before the thread refresh delay, they are as soon as possible applied.
• assertions: support library for assertions.
• serialization: here all the structures and components needed to (de-)serialize data managed by the BrokerService.
• planbuilders: once a requester asks the BrokerService to compile a deploy plan a PlanBuilderWorkflow is instantiated. For a sake of modularity, a workflow is built up to a chain of PlanBuilderTask elements each of one is designed to accomplish a specific goal (e.g. take the suggested GHNs that come with the request, check consistence of request...). The response to send to the requester is the result of these steps or, in case of failure, the representation of the PlanBuilderException raised during internal computation (e.g. the required GHNs are not defined on the IS or are locked by other plans). All the possible error messages that can be thrown are grouped inside the PlanExceptionMessages enum.

GetPlan

The entry point of ResourceBroker service is the getPlan operation. In Figure 2 is given a sketch of the internal logics (some aspects such as the interaction with the IS have been omitted for simplicity). Here is given an outline of the main implementation choices took in the getPlan.

• The plan requester (VREManager) sends a PlanRequest (whose structure is given below) to the ResourceBroker.
• The ResourceBroker retrieves the scope in which the deployment should be done (this information is contained inside the PlanRequest).
• In a such scope are instantiated the threads and the notification handlers responsible to monitor changes to the GHNs defined at such level (at IS side). Notice that a single instance of such components is done per scope. So further requests of creations of threads and notifications handlers for an already monitored scope are ignored.
• Now the plan request can be served. A new instance of plan workflow is created and associated to the current request.

Figure 2. getPlan.

Interface

Requesting a Deployment plan

<?xml version="1.0" ?> <PlanRequest xmlns="http://gcube-system.org/namespaces/resourcebroker/broker/xsd/deployRequest"> <Scope>/gcube/devsec</Scope> <PackageGroups> <PackageGroup> <Package reuse="true"> <ServiceClass>PkgServiceClass</ServiceClass> <ServiceName>PkgServiceName</ServiceName> <ServiceVersion>PkgServiceVersion</ServiceVersion> <PackageName>PkgPackageName</PackageName> <PackageVersion>PkgPackageVersion</PackageVersion> </Package> <Package reuse="true"> <ServiceClass>PkgServiceClass2</ServiceClass> <ServiceName>PkgServiceName2</ServiceName> <ServiceVersion>PkgServiceVersion2</ServiceVersion> <PackageName>PkgPackageName2</PackageName> <PackageVersion>PkgPackageVersion2</PackageVersion> </Package> <GHN>d5a5af20-ac50-11de-a928-ab32081f9f00</GHN> </PackageGroup> <PackageGroup> <Package reuse="true"> <ServiceClass>PkgServiceClass3</ServiceClass> <ServiceName>PkgServiceName3</ServiceName> <ServiceVersion>PkgServiceVersion3</ServiceVersion> <PackageName>PkgPackageName3</PackageName> <PackageVersion>PkgPackageVersion3</PackageVersion> </Package> </PackageGroup> <PackageGroup> <Package reuse="true"> <ServiceClass>PkgServiceClass3</ServiceClass> <ServiceName>PkgServiceName3</ServiceName> <ServiceVersion>PkgServiceVersion3</ServiceVersion> <PackageName>PkgPackageName3</PackageName> <PackageVersion>PkgPackageVersion3</PackageVersion> </Package> </PackageGroup> <PackageGroup> <Package reuse="true"> <ServiceClass>PkgServiceClass3</ServiceClass> <ServiceName>PkgServiceName3</ServiceName> <ServiceVersion>PkgServiceVersion3</ServiceVersion> <PackageName>PkgPackageName3</PackageName> <PackageVersion>PkgPackageVersion3</PackageVersion> </Package> </PackageGroup> <PackageGroup> <Package reuse="true"> <ServiceClass>PkgServiceClass3</ServiceClass> <ServiceName>PkgServiceName3</ServiceName> <ServiceVersion>PkgServiceVersion3</ServiceVersion> <PackageName>PkgPackageName3</PackageName> <PackageVersion>PkgPackageVersion3</PackageVersion> </Package> </PackageGroup> </PackageGroups> <GHNList> <GHN>1f251d60-215f-11df-b9c5-8ea2ab6d6650</GHN> <GHN>09ee4a70-1723-11df-adce-80cd45adc17d</GHN> </GHNList> </PlanRequest>