Revision as of 12:59, 4 May 2016

The integration of the gCube system with FedCloud and, in general, with research/commercial cloud providers, aims to add elasticity to the D4Science infrastructure by allowing programmatic and dynamic creation, configuration and decommissioning of cloud resources on external infrastructures. The benefits of this integration facility for the gCube system are threefold:

• to exploit infrastructural resources in a pay-per-use model
• to scale in/out quickly in response to specific needs
• to reduce manual interventions of Infrastructure Managers.

Scenarios

• the VRE Manager, according to the demand of data analysis executions in the VRE, decides the number of SmartExecutor nodes needed in the VRE. Through the VRE administration portal, the VRE Manager chooses (from a list of templates) to create the required number of SmartExecutors. She/he chooses on which infrastructure the resources should be created and their characteristics in terms of computation and/or storage capacity. The gCube system, via the integration facility here described, will automatically create the new nodes, register them to the D4Science infrastructure and make them available in the VRE.
• the gCube system intelligently adapt the number of SmartExecutor nodes to the actual demand. In the VRE administration portal, the VRE Manager decides the minimum and maximum number of SmartExecutor expected for the VRE. Then, the gCube system via the integration facility here described, automatically scales-in/out the pool of SmartExecutors (by creating/destroying nodes) predicting the demand for SmartExecutors by analysing VRE accounting and monitoring data.

Functional requirements

It is possible to recognize the following functionalities that should be available in the integration facilities:

• Registration and management of cloud providers, access credentials and cloud resources templates on a per-VRE basis. Credentials must be securely stored in the infrastructure;
• Cloud resource lifecycle management. Creation, configuration and decommissioning of cloud resources must be supported. The integration facility allows to use templates for the configuration of cloud resources (e.g. SmartExecutor Template, cpu and memory characteristics);
• Accountability and monitorability. Accounting and monitoring data on usage and workload of resources created on the cloud must be collected and made available in the infrastructure. This is both a requisite to comply with D4Science infrastructure policies and for the automatic scale-in/out;
• User interface. The user interface is integrated in the VRE administration portal since it already contains all the other VRE administration tools

Non-Functional requiriments

The solution should have modular architecture in order to support the integration of new cloud providers in a simple and pluggable way.

Architecture

The architecture highlights a number of integration-specific components and their interaction among them as well as with existing components, either belonging to the D4Science infrastructure or to the EGI infrastructure. They are described in the subsections above.

Cloud Libraries

Connectors

Federated Hosting Node Manager (FHNM)

Federated Hosting Node Manager Portlet

D4Science Information System

D4Science monitoring system

The Virtual Organization Membership Service (VOMS)

EGI FedCloud sites and AppDB

Deployment

Components related to the D4Science-FedCloud integration are essentially assembled in two packages that are deployed independently:

• the Federated Hosting Node Manager, embedding cloud libraries and connectors, is deployed in a SmartGears container
• the FHNManagerPortlet is deployed in the gCube portal.

Service Description

The solution has been developed by means of Java since it is the same technology of the gCube system and it made easier to integrate in the D4Science infrastructure and furthermore there is an high availability for third-party libraries implementing cloud standards.

A brief description of the service is visible in the following subsections.

Data Model

FHN Manager

FHN Manager Portlet

Third-party dependencies

EGI FedCloud membership

Source Code

The source code of the developed components are available on the gCube code repository.

Datamodel and Common: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-api/

Connector: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-occi-connector/

FHNManager: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-service/

FHN Client Library: http://svn.research-infrastructures.eu/d4science/gcube/trunk/vo-management/fhn-manager-client/

FHNManagerPortlet: http://svn.research-infrastructures.eu/d4science/gcube/trunk/portlets/admin/fhn-manager-portlet/

gCube release and testing process

Further documentation

User documentation for the FHNManagerPortlet can be found on the gCube User Guide: https://wiki.gcube-system.org/gcube/User%27s_Guide

FHNManager, connectors and libraries’ API documentation can be found on the gCube Developer’s Guide: https://wiki.gcube-system.org/gcube/Developer%27s_Guide

Deployment and configuration details for the service and the portlet are available on the gCube Administrator’s Guide: https://wiki.gcube-system.org/gcube/Administrator%27s_Guide

License

The FHNManagerPortlet, the FHNManager and the connectors developed within the scope of this task are released under the terms of the European Union Public Licence (EUPL version 1.1), like the rest of the gCube software.